AshGraphql filter for atom attribute stopped working in Ash 3

Ash Forum Ash Questions / Help
,
1

I have this attribute in my resource:

    attribute :status, :atom do
      allow_nil? false
      public? true

      constraints one_of: [:draft, :open, :pending, :sold, :inactive]

      default :draft
    end

And I have a read action that I call via AshGraphql to retrieve data from that resource.

When I try to filter that call by the status attribute, I’m getting an error.

Here is my graphql query:

query {
  listValidProperties(
    filter: {
      status: {in: ["OPEN", "PENDING", "SOLD"]}
    }
    limit: 5
    offset: 0
  ) {
    results {
      id
    }
  }
}

I get the following error in the terminal:

[warning] `96e7c8cc-071b-4623-a8a3-cad0b578f02f`: AshGraphql.Error not implemented for error:

** (Ash.Error.Query.InvalidFilterValue) Invalid filter value `status in ["OPEN", "PENDING", "SOLD"]`: No matching types. Possible types: [[:any, {:array, :same}]]
    (elixir 1.17.2) lib/process.ex:864: Process.info/2
    (ash 3.4.8) lib/ash/error/query/invalid_filter_value.ex:5: Ash.Error.Query.InvalidFilterValue.exception/1
    (ash 3.4.8) lib/ash/query/operator/operator.ex:184: Ash.Query.Operator.try_cast_with_ref/3
    (ash 3.4.8) lib/ash/filter/filter.ex:4015: anonymous fn/5 in Ash.Filter.parse_predicates/3
    (elixir 1.17.2) lib/enum.ex:4858: Enumerable.List.reduce/3
    (elixir 1.17.2) lib/enum.ex:2585: Enum.reduce_while/3
    (ash 3.4.8) lib/ash/filter/filter.ex:2764: Ash.Filter.add_expression_part/3
    (ash 3.4.8) lib/ash/filter/filter.ex:2902: anonymous fn/3 in Ash.Filter.add_expression_part/3
    (elixir 1.17.2) lib/enum.ex:4858: Enumerable.List.reduce/3
    (elixir 1.17.2) lib/enum.ex:2585: Enum.reduce_while/3
    (ash 3.4.8) lib/ash/filter/filter.ex:2901: Ash.Filter.add_expression_part/3
    (ash 3.4.8) lib/ash/filter/filter.ex:2460: anonymous fn/3 in Ash.Filter.parse_expression/2
    (elixir 1.17.2) lib/enum.ex:4858: Enumerable.List.reduce/3
    (elixir 1.17.2) lib/enum.ex:2585: Enum.reduce_while/3
    (ash 3.4.8) lib/ash/filter/filter.ex:334: Ash.Filter.parse/3
    (ash 3.4.8) lib/ash/query/query.ex:2699: Ash.Query.do_filter/3
    (stdlib 6.0.1) maps.erl:860: :maps.fold_1/4
    (ash_graphql 1.3.4) lib/graphql/resolver.ex:423: AshGraphql.Graphql.Resolver.resolve/2
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:234: Absinthe.Phase.Document.Execution.Resolution.reduce_resolution/1
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:189: Absinthe.Phase.Document.Execution.Resolution.do_resolve_field/3
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:174: Absinthe.Phase.Document.Execution.Resolution.do_resolve_fields/6
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:145: Absinthe.Phase.Document.Execution.Resolution.resolve_fields/4
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:88: Absinthe.Phase.Document.Execution.Resolution.walk_result/5
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:67: Absinthe.Phase.Document.Execution.Resolution.perform_resolution/3
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:24: Absinthe.Phase.Document.Execution.Resolution.resolve_current/3
    (absinthe 1.7.8) lib/absinthe/pipeline.ex:408: Absinthe.Pipeline.run_phase/3
    (absinthe_plug 1.5.8) lib/absinthe/plug.ex:536: Absinthe.Plug.run_query/4
    (absinthe_plug 1.5.8) lib/absinthe/plug.ex:290: Absinthe.Plug.call/2
    (phoenix 1.7.14) lib/phoenix/router/route.ex:42: Phoenix.Router.Route.call/2
    (phoenix 1.7.14) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint.plug_builder_call/2
    (core 1.91.3) deps/plug/lib/plug/debugger.ex:136: CoreWeb.Endpoint."call (overridable 3)"/2
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint."call (overridable 4)"/2
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint.call/2
    (phoenix 1.7.14) lib/phoenix/endpoint/sync_code_reload_plug.ex:22: Phoenix.Endpoint.SyncCodeReloadPlug.do_call/4
    (bandit 1.5.7) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2
    (bandit 1.5.7) lib/bandit/pipeline.ex:36: Bandit.Pipeline.run/4
    (bandit 1.5.7) lib/bandit/http1/handler.ex:12: Bandit.HTTP1.Handler.handle_data/3
    (bandit 1.5.7) lib/bandit/delegating_handler.ex:18: Bandit.DelegatingHandler.handle_data/3
    (bandit 1.5.7) /var/home/sezdocs/projects/rebuilt/platform/core/deps/thousand_island/lib/thousand_island/handler.ex:411: Bandit.DelegatingHandler.handle_continue/2
    (stdlib 6.0.1) gen_server.erl:2163: :gen_server.try_handle_continue/3
    (stdlib 6.0.1) gen_server.erl:2072: :gen_server.loop/7
    (stdlib 6.0.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3

And the following error as a response in graphql:

{
  "data": {
    "listValidProperties": null
  },
  "errors": [
    {
      "message": "Something went wrong. Unique error id: `96e7c8cc-071b-4623-a8a3-cad0b578f02f`",
      "path": [
        "listValidProperties"
      ],
      "locations": [
        {
          "line": 2,
          "column": 2
        }
      ]
    }
  ]
}

This used to work in Ash2 btw

2

Another thing that I noticed is that in Ash2, I was able to create “empty” filters like this:

query {
  listValidProperties(
    filter: {
      bedrooms: {}
    }
    limit: 5
    offset: 0
  ) {
    results {
      id
    }
  }
}

Where bedrooms is an integer attribute.

This query would work in Ash2, but in Ash3 it gives me the following error in the terminal:

[warning] `6e9b2ba9-0181-4ade-b87a-d3cba8477c99`: AshGraphql.Error not implemented for error:

** (Ash.Error.Query.InvalidFilterValue) Invalid filter value `true` supplied in `#Ecto.Query<from p0 in Core.Marketplace.Markets.Property, as: 0, join: o1 in ^#Ecto.Query<from o0 in Core.Marketplace.Markets.Organization, as: 0>, as: 1, on: as(0).organization_id == o1.id, where: type(as(0).bedrooms, {:parameterized, {Ash.Type.Integer.EctoType, []}}) ==
  type(^true, {:parameterized, {Ash.Type.Integer.EctoType, []}}), where: type(as(1).id, {:parameterized, {Ash.Type.UUIDv7.EctoType, []}}) ==
  type(^"185e2073-baa8-48f7-ba7b-cc873104a528", {:parameterized, {Ash.Type.UUIDv7.EctoType, []}}), where: type(
  as(0).status,
  {:parameterized, {Ash.Type.Atom.EctoType, one_of: [:draft, :open, :pending, :sold, :inactive]}}
) not in ^["inactive", "draft"], order_by: [asc: as(0).id], limit: ^6, select: merge(struct(p0, [:id]), %{
  full_address:
    type(
      type(
        fragment(
          "(? || ?)",
          type(
            type(
              as(0).house_number,
              {:parameterized, {AshPostgres.Type.StringWrapper.EctoType, []}}
            ),
            :string
          ),
          type(
            fragment(
              "(? || ?)",
              type(type(^" ", :string), :string),
              type(
                fragment(
                  "(? || ?)",
                  type(
                    type(
                      as(0).street,
                      {:parameterized, {AshPostgres.Type.StringWrapper.EctoType, []}}
                    ),
                    :string
                  ),
                  type(
                    fragment(
                      "(? || ?)",
                      type(type(^" - ", :string), :string),
                      type(
                        fragment(
                          "(? || ?)",
                          type(
                            type(
                              as(0).city,
                              {:parameterized, {AshPostgres.Type.StringWrapper.EctoType, []}}
                            ),
                            :string
                          ),
                          type(
                            fragment(
                              "(? || ?)",
                              type(type(^", ", :string), :string),
                              type(
                                fragment(
                                  "(? || ?)",
                                  type(
                                    type(
                                      as(0).state,
                                      {:parameterized,
                                       {AshPostgres.Type.StringWrapper.EctoType, []}}
                                    ),
                                    :string
                                  ),
                                  type(
                                    fragment(
                                      "(? || ?)",
                                      type(type(^" - ", :string), :string),
                                      type(
                                        type(
                                          as(0).zip,
                                          {:parameterized,
                                           {AshPostgres.Type.StringWrapper.EctoType, []}}
                                        ),
                                        :string
                                      )
                                    ),
                                    :string
                                  )
                                ),
                                :string
                              )
                            ),
                            :string
                          )
                        ),
                        :string
                      )
                    ),
                    :string
                  )
                ),
                :string
              )
            ),
            :string
          )
        ),
        {:parameterized, {AshPostgres.Type.StringWrapper.EctoType, []}}
      ),
      {:parameterized, {AshPostgres.Type.StringWrapper.EctoType, []}}
    ),
  favorite?:
    type(
      type(
        exists(
          subquery(
            #Ecto.Query<from p0 in Core.Marketplace.Markets.PropertyUserActivity, as: 3, where: type(as(3).user_id, {:parameterized, {Ash.Type.UUIDv7.EctoType, []}}) ==
  type(^nil, {:parameterized, {Ash.Type.UUIDv7.EctoType, []}}), where: type(as(3).favorited?, {:parameterized, {Ash.Type.Boolean.EctoType, []}}), where: parent_as(0).id == p0.property_id, select: 1>
          )
        ),
        {:parameterized, {Ash.Type.Boolean.EctoType, []}}
      ),
      {:parameterized, {Ash.Type.Boolean.EctoType, []}}
    )
})>`
    (elixir 1.17.2) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.17.2) lib/enum.ex:1829: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir 1.17.2) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.12.3) lib/ecto/repo/queryable.ex:214: Ecto.Repo.Queryable.execute/4
    (ecto 3.12.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (ash_postgres 2.3.1) lib/data_layer.ex:769: anonymous fn/3 in AshPostgres.DataLayer.run_query/2
    (ash_postgres 2.3.1) lib/data_layer.ex:767: AshPostgres.DataLayer.run_query/2
    (ash 3.4.8) lib/ash/actions/read/read.ex:2598: Ash.Actions.Read.run_query/4
    (ash 3.4.8) lib/ash/actions/read/read.ex:517: anonymous fn/7 in Ash.Actions.Read.do_read/4
    (ash 3.4.8) lib/ash/actions/read/read.ex:861: Ash.Actions.Read.maybe_in_transaction/3
    (ash 3.4.8) lib/ash/actions/read/read.ex:264: Ash.Actions.Read.do_run/3
    (ash 3.4.8) lib/ash/actions/read/read.ex:81: anonymous fn/3 in Ash.Actions.Read.run/3
    (ash 3.4.8) lib/ash/actions/read/read.ex:80: Ash.Actions.Read.run/3
    (ash 3.4.8) lib/ash.ex:1975: Ash.read/2
    (ash_graphql 1.3.4) lib/graphql/resolver.ex:474: AshGraphql.Graphql.Resolver.resolve/2
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:234: Absinthe.Phase.Document.Execution.Resolution.reduce_resolution/1
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:189: Absinthe.Phase.Document.Execution.Resolution.do_resolve_field/3
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:174: Absinthe.Phase.Document.Execution.Resolution.do_resolve_fields/6
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:145: Absinthe.Phase.Document.Execution.Resolution.resolve_fields/4
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:88: Absinthe.Phase.Document.Execution.Resolution.walk_result/5
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:67: Absinthe.Phase.Document.Execution.Resolution.perform_resolution/3
    (absinthe 1.7.8) lib/absinthe/phase/document/execution/resolution.ex:24: Absinthe.Phase.Document.Execution.Resolution.resolve_current/3
    (absinthe 1.7.8) lib/absinthe/pipeline.ex:408: Absinthe.Pipeline.run_phase/3
    (absinthe_plug 1.5.8) lib/absinthe/plug.ex:536: Absinthe.Plug.run_query/4
    (absinthe_plug 1.5.8) lib/absinthe/plug.ex:290: Absinthe.Plug.call/2
    (phoenix 1.7.14) lib/phoenix/router/route.ex:42: Phoenix.Router.Route.call/2
    (phoenix 1.7.14) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint.plug_builder_call/2
    (core 1.91.3) deps/plug/lib/plug/debugger.ex:136: CoreWeb.Endpoint."call (overridable 3)"/2
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint."call (overridable 4)"/2
    (core 1.91.3) lib/core_web/endpoint.ex:1: CoreWeb.Endpoint.call/2
    (phoenix 1.7.14) lib/phoenix/endpoint/sync_code_reload_plug.ex:22: Phoenix.Endpoint.SyncCodeReloadPlug.do_call/4
    (bandit 1.5.7) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2
    (bandit 1.5.7) lib/bandit/pipeline.ex:36: Bandit.Pipeline.run/4
    (bandit 1.5.7) lib/bandit/http1/handler.ex:12: Bandit.HTTP1.Handler.handle_data/3
    (bandit 1.5.7) lib/bandit/delegating_handler.ex:18: Bandit.DelegatingHandler.handle_data/3
    (bandit 1.5.7) /var/home/sezdocs/projects/rebuilt/platform/core/deps/thousand_island/lib/thousand_island/handler.ex:379: Bandit.DelegatingHandler.handle_info/2
    (stdlib 6.0.1) gen_server.erl:2173: :gen_server.try_handle_info/3
    (stdlib 6.0.1) gen_server.erl:2261: :gen_server.handle_msg/6
    (stdlib 6.0.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3

Adn the following error in the graphql:

{
  "data": {
    "listValidProperties": null
  },
  "errors": [
    {
      "message": "Something went wrong. Unique error id: `6e9b2ba9-0181-4ade-b87a-d3cba8477c99`",
      "path": [
        "listValidProperties"
      ],
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ]
    }
  ]
}
3

See the upgrading guide in AshGraphql for more on this. We no longer derive named types for inline attributes like this. Upgrading to 1.0 — ash_graphql v1.3.4

For the other one, please open an issue in ash. I think it should be reproducible by passing exactly that filter to filter_input, i.e

Resource
|> Ash.Query.filter_input(%{bedrooms: %{}})

Can you see if that is the case?

4

Ok, actually, I think you can ignore that second issue I posted.

My front dev that said that this used to work in Ash2, but I just tried it manually and it gives the same error.

So it is not a regression, it is just working as it should, I will request him to change the frontend to not generate these broken filters anymore.

5

We should still fix them. There shouldn’t really be any filter structure like that that just “fails” without giving some reasonable error explaining how to fix it.

6

I will create an issue ASAP then

7

@zachdaniel just one more quick question about this.

I think I will go the explicit route and create one field_policy per attribute the same way that I do with one single policy per action, but I do remember that in Ash2, when ash_rbac was being developed, there was some performance issues when creating a bunch of policies.

Is this still an issue or is it fine to create a bunch of field_policies? I do have some resources that have more than 250 attributes.

8

it should be fine, but honestly i’d test it and see.

1 Like
9

@zachdaniel is there some call that I can do to just run the field_policies for a resource action without running the full action (like we can use Ash.can? for normal policies)?

Basically I’m creating some benchmarks to test if there is any impact in performance, but it would be great if I can shrink the scope just to the field policies.

10

Sort of. You can fetch a record with select([]) and then Ash.load(thing, [:your, :big, :list, :of :attributes], actor: actor). That tests a bit more than just field policies, but shouldn’t go to the database or anything.

11

Hmm, why would that not go to the database? Shouldn’t it fetch all these fields in the load call? At least that’s what I’m seeing here when I tried it.

Here is my code:

records = Record |> Ash.Query.for_read(:read, %{}) |> Ash.Query.select([]) |> Ash.Query.limit(100) |>  Ash.read!()

fields = [,,,]

Ash.load!(records, fields, actor: actor)
12

Ah, yeah you’re right :slight_smile: Wasn’t thinking. I don’t think there is a good way to just apply field policies TBH. They are implemented as calculations.

13

I’m gonna try using the ets datalayer since I think that one would be more consistent on timing.

1 Like
14

So, @zachdaniel , I did some benchmark testing here and I think I got some interesting numbers.

The test consists of a resource that contains around 255 attributes. I used the ETS data-layer and added one record to it.

Then, I run a read action to fetch it applying my field policies and another run not applying it.

The benchmark code is as follows:

defmodule Benchmark do
  alias Core.Pacman.Markets.Record
  
  def run() do
    actor = %{roles: [:blibs]}

    Benchee.run(
      %{
        "run" => fn ->
          Record
          |> Ash.Query.for_read(:read, %{}, actor: actor)
          |> Ash.read!()
        end,
      },
      time: 100,
      memory_time: 2
    )
  end
end

For the field policies, I added one per attribute, all of them have the same check, I did two types of checks to see if the check itself would be a bottleneck.

Using a check that search for a atom in the :roles field of an agent (based on the HasRole check from ash rbac):

      field_policy :external_id do
        authorize_if {HasRole, role: [:support, :admin, {:agent, :organization_roles}]}
      end

Using a always authorize check:

      field_policy :external_id do
        authorize_if always()
      end

Here are the results:

Name               ips        average  deviation         median         99th %
disabled       4885.16        0.20 ms    ±16.71%       0.197 ms        0.34 ms
always           50.60       19.76 ms    ±19.98%       19.15 ms       30.24 ms
roles            36.92       27.09 ms    ±11.88%       26.44 ms       38.14 ms

Comparison: 
disabled       4885.16
always           50.60 - 96.55x slower +19.56 ms
roles            36.92 - 132.33x slower +26.88 ms

Memory usage statistics:

Name             average  deviation         median         99th %
disabled       0.0254 MB     ±0.01%      0.0254 MB      0.0254 MB
always           5.22 MB     ±0.06%        5.22 MB        5.23 MB
roles           15.31 MB     ±0.06%       15.31 MB       15.34 MB

Comparison: 
disabled       0.0254 MB
always           5.22 MB - 205.84x memory usage +5.20 MB
roles           15.31 MB - 603.57x memory usage +15.29 MB

Seems like there is a very big overhead of creating field policies per field

15

I just added some more benchmark numbers:

Name                       ips        average  deviation         median         99th %
disabled               4885.16        0.20 ms    ±16.71%       0.197 ms        0.34 ms
catch_all_always       1227.80        0.81 ms    ±22.45%        0.75 ms        1.61 ms
catch_all_roles         964.08        1.04 ms    ±13.55%        1.00 ms        1.57 ms
always                   50.60       19.76 ms    ±19.98%       19.15 ms       30.24 ms
bypass_roles             37.81       26.45 ms    ±13.31%       26.03 ms       35.65 ms
roles                    36.92       27.09 ms    ±11.88%       26.44 ms       38.14 ms

Comparison: 
disabled               4885.16
catch_all_always       1227.80 - 3.98x slower +0.61 ms
catch_all_roles         964.08 - 5.07x slower +0.83 ms
always                   50.60 - 96.55x slower +19.56 ms
bypass_roles             37.81 - 129.19x slower +26.24 ms
roles                    36.92 - 132.33x slower +26.88 ms

Memory usage statistics:

Name                     average  deviation         median         99th %
disabled               0.0254 MB     ±0.01%      0.0254 MB      0.0254 MB
catch_all_always        0.142 MB     ±0.00%       0.142 MB       0.142 MB
catch_all_roles          0.29 MB     ±0.01%        0.29 MB        0.29 MB
always                   5.22 MB     ±0.06%        5.22 MB        5.23 MB
bypass_roles            15.10 MB     ±0.13%       15.10 MB       15.15 MB
roles                   15.31 MB     ±0.06%       15.31 MB       15.34 MB

Comparison: 
disabled               0.0254 MB
catch_all_always        0.142 MB - 5.62x memory usage +0.117 MB
catch_all_roles          0.29 MB - 11.45x memory usage +0.27 MB
always                   5.22 MB - 205.84x memory usage +5.20 MB
bypass_roles            15.10 MB - 595.18x memory usage +15.07 MB
roles                   15.31 MB - 603.57x memory usage +15.29 MB

Legend:
disabled: field_policies are fully disabled
catch_all_always: there is only one field_policy with a catch_all that always authorize
catch_all_roles: there is only one field_policy with a catch_all that checks for the role
always: have a field_policy per attribute that always authorize
bypass_roles: have a field_policy_bypass per attribute that will check for the role
roles: have a field_policy per attribute that will check for the role

It is interesting to see that even having just one field_policy already makes the query 4x slower than not having any check at all

16

Yeah, I’d imagine this area is rife for optimizations :slight_smile:

Ultimately, I think having fewer field policies is better than one policy per field from a conceptual standpoint, and we should come up with a way for you to write either a deny-list or an allow-list.

I’d be willing to bet that we could eliminate most if not all of that additional processing time via optimizing those code paths. If you could open an issue on Ash showing this info that would be great. I won’t have the time to optimize this in the near future, unfortunately, but perhaps an adventurous soul can investigate.

17

Would you mind giving me some hints which files/functions the field policies code path is run? I can take a look into it when I have some free time in my hands.

18

The add_calculations callback is where we determine the calculations, and then in add_field_level_auth function in the read action logic.

1 Like