Count aggregation doesn´t seem to work on no_attribute relationships

I have the following count aggregate in my resource:

count :prior_year_transactions, :owner_records

in the same resource, the :owner_records relationship is defined as such:

    has_many :owner_records, Record do
      no_attributes? true

      read_action :valid

      filter expr(
            fragment("(? %> ?)", first_grantee_full_name, parent(first_owner[:full_name]))
            )
    end

It uses the :valid read action from the Record resource with is defined as such:

    read :valid do
      filter expr(
                 not is_nil(recording_date || instrument_date) and
                 (recording_date || instrument_date) >= ago(2, :year) and
                 not deleted?
             )

      prepare build(sort: [:recording_date], load: [:sell_record, :fix_and_flip?])
    end

When I try to load that aggregate, I get the following error:

** (Ash.Error.Unknown) Unknown Error

* ** (Ecto.SubQueryError) the following exception happened when compiling a subquery.

    ** (Ecto.QueryError) deps/ash_postgres/lib/aggregate.ex:1049: field `property_id` in `select` does not exist in schema Pacman.Raw.Record in query:
    
    from r0 in Pacman.Raw.Record,
      as: 0,
      where: fragment(
      "(? %> ?)",
      type(
        as(0).first_grantee_full_name,
        {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
         max_length: 255}
      ),
      type(
        fragment(
          "jsonb_extract_path_text(?::jsonb,?::text)",
          type(
            parent_as(0).first_owner,
            {:parameterized, Pacman.Markets.Property.Owner.EctoType, []}
          ),
          ^"full_name"
        ),
        {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
         max_length: 255}
      )
    ),
      where: parent_as(0).id == as(0).property_id,
      group_by: [r0.property_id],
      select: %{
      prior_year_transactions:
        type(
          coalesce(
            count(type(as(0).id, {:parameterized, Ash.Type.UUID.EctoType, []})),
            type(^0, {:parameterized, Ash.Type.Integer.EctoType, []})
          ),
          {:parameterized, Ash.Type.Integer.EctoType, []}
        ),
      property_id: map(r0, [:property_id]).property_id
    }
    

The subquery originated from the following query:

from p0 in Pacman.Markets.Property,
  as: 0,
  left_lateral_join: r1 in subquery(from r0 in Pacman.Raw.Record,
  as: 0,
  where: fragment(
  "(? %> ?)",
  type(
    as(0).first_grantee_full_name,
    {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
     max_length: 255}
  ),
  type(
    fragment(
      "jsonb_extract_path_text(?::jsonb,?::text)",
      type(
        parent_as(0).first_owner,
        {:parameterized, Pacman.Markets.Property.Owner.EctoType, []}
      ),
      ^"full_name"
    ),
    {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
     max_length: 255}
  )
),
  where: parent_as(0).id == as(0).property_id,
  group_by: [r0.property_id],
  select: merge(
  %{
    prior_year_transactions:
      type(
        coalesce(
          count(type(as(0).id, {:parameterized, Ash.Type.UUID.EctoType, []})),
          type(^0, {:parameterized, Ash.Type.Integer.EctoType, []})
        ),
        {:parameterized, Ash.Type.Integer.EctoType, []}
      )
  },
  map(r0, [:property_id])
)),
  as: 1,
  on: true,
  where: type(
  as(0).external_id,
  {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
   max_length: 255}
) ==
  type(
    ^"34025875",
    {:parameterized, Ash.Type.StringWrapper.EctoType, allow_empty?: false, trim?: true,
     max_length: 255}
  ),
  where: type(as(0).external_id_type, {:parameterized, Ash.Type.Atom.EctoType, one_of: [:attom_id]}) ==
  type(^"attom_id", {:parameterized, Ash.Type.Atom.EctoType, one_of: [:attom_id]}),
  select: merge(
  struct(p0, [
    :id,
    :external_id,
    :external_id_type,
    :cbsa_code,
    :msa_code,
    :minor_civil_division_code,
    :full_address,
    :house_number,
    :street_direction,
    :street_post_direction,
    :street_name,
    :street_suffix,
    :unit_prefix,
    :unit_value,
    :city,
    :county,
    :county_fips,
    :state,
    :zip,
    :zip_4,
    :country,
    :geography,
    :year_built,
    :lot_square_feet,
    :lot_depth,
    :lot_width,
    :gross_area,
    :living_area,
    :garage_area,
    :building_area,
    :baths,
    :partial_baths,
    :bedrooms,
    :rooms,
    :units,
    :stories,
    :parking_spaces,
    :zoning_code,
    :type,
    :roof_type,
    :construction_type,
    :air_conditioning_type,
    :garage_types,
    :basement_type,
    :water_source,
    :home_owners_association_type,
    :multiple_listing_service_status,
    :property_classification,
    :last_sale,
    :mortgage,
    :assessment,
    :tax,
    :first_owner,
    :second_owner,
    :contact,
    :acquisition_platform,
    :publication_date,
    :upload_date,
    :attom_transaction_time,
    :ap_transaction_time,
    :deleted?,
    :normalized_full_address,
    :occupancy,
    :inserted_at,
    :updated_at
  ]),
  %{
    prior_year_transactions:
      type(
        coalesce(
          type(as(1).prior_year_transactions, {:parameterized, Ash.Type.Integer.EctoType, []}),
          type(^0, {:parameterized, Ash.Type.Integer.EctoType, []})
        ),
        {:parameterized, Ash.Type.Integer.EctoType, []}
      )
  }
)

  (ecto 3.10.3) lib/ecto/repo/queryable.ex:211: Ecto.Repo.Queryable.execute/4
  (ecto 3.10.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
  (ash_postgres 1.3.60) lib/data_layer.ex:645: AshPostgres.DataLayer.run_query/2
  (ash 2.17.1) lib/ash/actions/read.ex:2886: Ash.Actions.Read.run_query/6
  (ash 2.17.1) lib/ash/actions/read.ex:1448: anonymous fn/4 in Ash.Actions.Read.data_field/3
  (ash 2.17.1) lib/ash/engine/engine.ex:537: anonymous fn/2 in Ash.Engine.run_iteration/1
  (ash 2.17.1) lib/ash/engine/engine.ex:558: anonymous fn/4 in Ash.Engine.async/2
  (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
  (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
    (ash 2.17.1) lib/ash/error/error.ex:524: Ash.Error.choose_error/2
    (ash 2.17.1) lib/ash/error/error.ex:252: Ash.Error.to_error_class/2
    (ash 2.17.1) lib/ash/actions/read.ex:211: Ash.Actions.Read.do_run/3
    (ash 2.17.1) lib/ash/actions/read.ex:97: anonymous fn/3 in Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/actions/read.ex:96: Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/api/api.ex:2000: Ash.Api.read!/3

I’m not sure what I’m doing wrong of if this aggregate doesn’t work well with relationships that uses no_attributes?

I’m actually investigating a similar issue currently, will get back to you :slight_smile:

Let me know if I can do something to help you out on the investigation :slight_smile:

Can you try ash_postgres main branch?

Just tested it, now I get the following error:

iex(3)> Pacman.Markets.Property |> Ash.Query.new() |> Ash.Query.for_read(:get_by_attom_id, %{attom_id: "34025875"}) |> Ash.Query.load([:prior_year_transactions]) |> Pacman.Markets.read!()
[debug] QUERY ERROR source="properties" db=0.0ms queue=0.7ms idle=1227.9ms
SELECT p0."id", p0."external_id", p0."external_id_type", p0."cbsa_code", p0."msa_code", p0."minor_civil_division_code", p0."full_address", p0."house_number", p0."street_direction", p0."street_post_direction", p0."street_name", p0."street_suffix", p0."unit_prefix", p0."unit_value", p0."city", p0."county", p0."county_fips", p0."state", p0."zip", p0."zip_4", p0."country", p0."geography", p0."year_built", p0."lot_square_feet", p0."lot_depth", p0."lot_width", p0."gross_area", p0."living_area", p0."garage_area", p0."building_area", p0."baths", p0."partial_baths", p0."bedrooms", p0."rooms", p0."units", p0."stories", p0."parking_spaces", p0."zoning_code", p0."type", p0."roof_type", p0."construction_type", p0."air_conditioning_type", p0."garage_types", p0."basement_type", p0."water_source", p0."home_owners_association_type", p0."multiple_listing_service_status", p0."property_classification", p0."last_sale", p0."mortgage", p0."assessment", p0."tax", p0."first_owner", p0."second_owner", p0."contact", p0."acquisition_platform", p0."publication_date", p0."upload_date", p0."attom_transaction_time", p0."ap_transaction_time", p0."deleted?", p0."normalized_full_address", p0."occupancy", p0."inserted_at", p0."updated_at", coalesce(s1."prior_year_transactions"::bigint, $1::bigint)::bigint FROM "properties" AS p0 LEFT OUTER JOIN LATERAL (SELECT coalesce(count(sr0."id"::uuid), $2::bigint)::bigint AS "prior_year_transactions" FROM "raw"."records" AS sr0 WHERE ((sr0."first_grantee_full_name"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,$3::text)::text)) AND (p0."id" = sr0."property_id")) AS s1 ON TRUE WHERE (p0."external_id"::text = $4::text) AND (p0."external_id_type"::varchar = $5::varchar) [0, 0, "full_name", "34025875", :attom_id]
** (Ash.Error.Unknown) Unknown Error

* ** (Postgrex.Error) ERROR 42703 (undefined_column) column sr0.property_id does not exist

    query: SELECT p0."id", p0."external_id", p0."external_id_type", p0."cbsa_code", p0."msa_code", p0."minor_civil_division_code", p0."full_address", p0."house_number", p0."street_direction", p0."street_post_direction", p0."street_name", p0."street_suffix", p0."unit_prefix", p0."unit_value", p0."city", p0."county", p0."county_fips", p0."state", p0."zip", p0."zip_4", p0."country", p0."geography", p0."year_built", p0."lot_square_feet", p0."lot_depth", p0."lot_width", p0."gross_area", p0."living_area", p0."garage_area", p0."building_area", p0."baths", p0."partial_baths", p0."bedrooms", p0."rooms", p0."units", p0."stories", p0."parking_spaces", p0."zoning_code", p0."type", p0."roof_type", p0."construction_type", p0."air_conditioning_type", p0."garage_types", p0."basement_type", p0."water_source", p0."home_owners_association_type", p0."multiple_listing_service_status", p0."property_classification", p0."last_sale", p0."mortgage", p0."assessment", p0."tax", p0."first_owner", p0."second_owner", p0."contact", p0."acquisition_platform", p0."publication_date", p0."upload_date", p0."attom_transaction_time", p0."ap_transaction_time", p0."deleted?", p0."normalized_full_address", p0."occupancy", p0."inserted_at", p0."updated_at", coalesce(s1."prior_year_transactions"::bigint, $1::bigint)::bigint FROM "properties" AS p0 LEFT OUTER JOIN LATERAL (SELECT coalesce(count(sr0."id"::uuid), $2::bigint)::bigint AS "prior_year_transactions" FROM "raw"."records" AS sr0 WHERE ((sr0."first_grantee_full_name"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,$3::text)::text)) AND (p0."id" = sr0."property_id")) AS s1 ON TRUE WHERE (p0."external_id"::text = $4::text) AND (p0."external_id_type"::varchar = $5::varchar)

    hint: Perhaps you meant to reference the column "sr0.property_zip".
  (ecto_sql 3.11.0) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
  (ecto_sql 3.11.0) lib/ecto/adapters/sql.ex:952: Ecto.Adapters.SQL.execute/6
  (ecto 3.11.0) lib/ecto/repo/queryable.ex:232: Ecto.Repo.Queryable.execute/4
  (ecto 3.11.0) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
  (ash_postgres 1.3.62) lib/data_layer.ex:646: AshPostgres.DataLayer.run_query/2
  (ash 2.17.1) lib/ash/actions/read.ex:2886: Ash.Actions.Read.run_query/6
  (ash 2.17.1) lib/ash/actions/read.ex:1448: anonymous fn/4 in Ash.Actions.Read.data_field/3
  (ash 2.17.1) lib/ash/engine/engine.ex:537: anonymous fn/2 in Ash.Engine.run_iteration/1
  (ash 2.17.1) lib/ash/engine/engine.ex:558: anonymous fn/4 in Ash.Engine.async/2
  (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
  (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
    (ash 2.17.1) lib/ash/error/error.ex:524: Ash.Error.choose_error/2
    (ash 2.17.1) lib/ash/error/error.ex:252: Ash.Error.to_error_class/2
    (ash 2.17.1) lib/ash/actions/read.ex:211: Ash.Actions.Read.do_run/3
    (ash 2.17.1) lib/ash/actions/read.ex:97: anonymous fn/3 in Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/actions/read.ex:96: Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/api/api.ex:2000: Ash.Api.read!/3

Seems like it is still trying to find a property_id field in the relationship even tought the relationship don’t actually have that because is fetches the data without a foreign key:

oops, can you try again?

It is not crashing anymore, but the result seem to be wrong.

The :owner_records fields has 8 records inside its list, but the count aggregate is returning 781.

here is the query that Ash made to my DB:

SELECT p0."id", p0."external_id", p0."external_id_type", p0."cbsa_code", p0."msa_code", p0."minor_civil_division_code", p0."full_address", p0."house_number", p0."street_direction", p0."street_post_direction", p0."street_name", p0."street_suffix", p0."unit_prefix", p0."unit_value", p0."city", p0."county", p0."county_fips", p0."state", p0."zip", p0."zip_4", p0."country", p0."geography", p0."year_built", p0."lot_square_feet", p0."lot_depth", p0."lot_width", p0."gross_area", p0."living_area", p0."garage_area", p0."building_area", p0."baths", p0."partial_baths", p0."bedrooms", p0."rooms", p0."units", p0."stories", p0."parking_spaces", p0."zoning_code", p0."type", p0."roof_type", p0."construction_type", p0."air_conditioning_type", p0."garage_types", p0."basement_type", p0."water_source", p0."home_owners_association_type", p0."multiple_listing_service_status", p0."property_classification", p0."last_sale", p0."mortgage", p0."assessment", p0."tax", p0."first_owner", p0."second_owner", p0."contact", p0."acquisition_platform", p0."publication_date", p0."upload_date", p0."attom_transaction_time", p0."ap_transaction_time", p0."deleted?", p0."normalized_full_address", p0."occupancy", p0."inserted_at", p0."updated_at", coalesce(s1."prior_year_transactions"::bigint, $1::bigint)::bigint, exists((SELECT 1 AS "result" FROM "raw"."records" AS sr0 WHERE ((sr0."first_grantee_full_name"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,$2::text)::text)))) FROM "properties" AS p0 LEFT OUTER JOIN LATERAL (SELECT coalesce(count(sr0."id"::uuid), $3::bigint)::bigint AS "prior_year_transactions" FROM "raw"."records" AS sr0 WHERE ((sr0."first_grantee_full_name"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,$4::text)::text))) AS s1 ON TRUE WHERE (p0."external_id"::text = $5::text) AND (p0."external_id_type"::varchar = $6::varchar) [0, "full_name", 0, "full_name", "34025875", :attom_id]

I will take a closer look into that query to see what is doing, but I wanted to give you the feedback first ASAP

@zachdaniel I think I understand what is happening here.

If we look into the left join in the above query. it will do this select inside:

SELECT coalesce(count(sr0."id"::uuid), 0::bigint)::bigint AS "prior_year_transactions"
FROM "raw"."records" AS sr0
WHERE ((sr0."first_grantee_full_name"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,'full_name'::text)::text))

As you can see, it is using the filter I added to the has_many block:

      filter expr(
            fragment("(? %> ?)", first_grantee_full_name, parent(first_owner[:full_name]))
            )

But it is totally ignoring the read_action :valid from the has_many and just using the primary :read action.

I think that the aggregate should use the relationship read action that was set , not the default one in this case.

I tried to add the :valid read action to the aggregate directly to see what would happend:

count :prior_year_transactions, :owner_records do
  read_action :valid
end

But this will fail with the following error:

** (Ash.Error.Unknown) Unknown Error

Context: Loading aggregate: :prior_year_transactions for query: #Ash.Query<resource: Pacman.Markets.Property, arguments: %{attom_id: "34025875"}, filter: #Ash.Filter<external_id == "34025875" and external_id_type == :attom_id>>
* Context: Loading aggregate: :prior_year_transactions for query: #Ash.Query<resource: Pacman.Markets.Property, arguments: %{attom_id: "34025875"}, filter: #Ash.Filter<external_id == "34025875" and external_id_type == :attom_id>>

Cannot load in an aggregate.
    at aggregates
  (ash 2.17.1) lib/ash/error/error.ex:391: Ash.Error.to_ash_error/3
  (ash 2.17.1) lib/ash/query/query.ex:1371: Ash.Query.do_load/2
  (elixir 1.15.7) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3
  (stdlib 5.1.1) erl_eval.erl:746: :erl_eval.do_apply/7
  (stdlib 5.1.1) erl_eval.erl:1026: :erl_eval.expr_list/7
    (ash 2.17.1) lib/ash/error/error.ex:524: Ash.Error.choose_error/2
    (ash 2.17.1) lib/ash/error/error.ex:252: Ash.Error.to_error_class/2
    (ash 2.17.1) lib/ash/actions/read.ex:211: Ash.Actions.Read.do_run/3
    (ash 2.17.1) lib/ash/actions/read.ex:97: anonymous fn/3 in Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/actions/read.ex:96: Ash.Actions.Read.run/3
    (ash 2.17.1) lib/ash/api/api.ex:2000: Ash.Api.read!/3

So. I’m not sure we can make this work the way that you want it to work TBH. The essential problem is that we could maybe build the target query for a relationship action, but we can’t actually run that target query. So it would get confusing how they compose. Like if there was a before action hook, it would never be run. I would suggest extracting the filter from the action into a named calculation, and using that in your aggregate to filter. That way the target resource is still the source of truth, and the way that it composes is clear.

To perhaps explain a bit better: read_action on relationships will be honored in full when loading related data, but when formulating aggregates, we can’t actually build/run the query. read_action is primarily used to customize the authorization logic behind loading an aggregate.

hey @zachdaniel , thanks for the help so far.

So, what I did was encapsulate some of the complexity of the query inside a view.

Doing that, now my read action for the Entity resource is:

    read :all do
      prepare build(sort: [buy_date: :desc])
    end

And for the Property resource, the relationship is:

    has_many :owner_records, Pacman.Raw.EntityRecord do
      no_attributes? true

      read_action :all

      filter expr(fragment("(? %> ?)", buyer, parent(first_owner[:full_name])))
    end

With that, here is my count aggregate:

    count :total_prior_year_transactions, :owner_records do
      filter expr(
             fragment("extract('year' from ?)", buy_date) ==
               fragment("extract('year' from now() - interval '1' year)")
             )

      field :external_id
    end

What I expect to receive here is the number 4, but I’m getting 143 for some reason.

Here is the SQL query Ash generates when loading this aggregate:

SELECT p0."id", coalesce(s1."total_prior_year_transactions"::bigint, 0::bigint)::bigint
FROM "properties" AS p0
LEFT OUTER JOIN LATERAL (
  SELECT coalesce(count(se0."external_id"::text), 0::bigint)::bigint AS "total_prior_year_transactions"
  FROM "raw"."entity_records" AS se0
  WHERE ((se0."buyer"::text %> jsonb_extract_path_text(p0."first_owner"::jsonb::jsonb,'full_name'::text)::text))
    AND (extract('year' from se0."buy_date"::date) = extract('year' from now() - interval '1' year))
) AS s1 ON TRUE
WHERE (p0."id"::uuid = '728440f3-0e23-4f85-ad58-f80df6b4240d'::uuid);

What is more curious is that, if I run this query by hand in psql, I get this result:

                  id                  | coalesce 
--------------------------------------+----------
 728440f3-0e23-4f85-ad58-f80df6b4240d |        4
(1 row)

Which is correct, i just don’t get why I’m getting that 143 number instead of the 4 the query is correctly returning.

Any guesses?

:thinking: That seems pretty strange. Not sure why you’d be getting a different number TBH. I’ve made some fixes to ash_postgres today, can you double check against the latest main?