How to use Pagination on Calculations? Do I need to handcraft my own Pagination macro?

[FedeMadoery]

I’m trying to do a pagination for a list of “messages” on a calculated field, so my structure is more or like the following one:

Channel Resource

defmodule Organizations.Channel do
  use Ash.Resource,
    data_layer: AshPostgres.DataLayer,
    extensions: [
      AshGraphql.Resource
    ]

  require Ash.Query

  # Some other configs ...
 
  relationships do
    has_many :messages, Channels.ChannelMessage do
      private? true
      destination_attribute :channel_id
      api Channels
  end 

  actions do
    defaults [:create, :update, :destroy]

    # This one creates a PageOfChannels
    read :list do
      pagination countable: true,
                 required?: true,
                 offset?: true,
                 max_page_size: 10
    end

    read :read do
      primary? true
    end
  end

Channel Messages Resource

defmodule Channels.ChannelMessage do
  use Ash.Resource,
    data_layer: AshPostgres.DataLayer,
    extensions: [
      AshGraphql.Resource
    ]

  # Some other configs ...

  actions do
    # This creates a  PageOfChannelMessages
    read :list do
      pagination countable: true,
                 required?: true,
                 offset?: true,
                 max_page_size: 10,
                 default_limit: 10
    end

    read :read do
      primary? true
    end
  end
  
  relationships do
    belongs_to :channel, Channels.Channel do
      allow_nil? false
      primary_key? true
    end
  end

With that said, I want to access my ChannelMessages, from the Channel resource, so I added a calculation for it, but I also want to paginate them using the same Type and structure (in this case PageOfChannelMessages) to share entities on the UI side with ApolloClient

So I did:

calculations do
    calculate :channel_messages, {:array, ChannelMessageUnion}, fn record, %{api: api} ->
      record = api.load!(record, :messages)

      record.messages
      |> Enum.map(&%Ash.Union{type: ChannelMessageUnion.struct_to_name(&1), value: &1})
    end

But with this calculation, my channel_messages is just an array with all the messages or just a trimmed array if I limit it, so I don’t see how I can calculate as a PageOf type since those types are not accessible or I don’t know how, so with this in mind, my question would be can I reuse the actual pagination of AshGraphQL o should I cook my own pagination to be able to re-use it across all my app (calculation and lists)

[zachdaniel]

I’ve been wanting to support pagination of related resources in ash core, but currently we don’t unfortunately. However, you should be able to achieve this using a custom type. I’m not sure this exact usage has been done before (making a custom map type that mimics an existing defined page type), so please LMK if you run into any issues.

First, define your custom type. I’m using a shorthand with Ash.Type.NewType to define a type based on :map, but that allows me to define graphql_type/1 callback.

defmodule MyApp.PageOfChannelMessages do
  use Ash.Type.NewType, subtype_of: :map
  use AshGraphql.Type

  def graphql_type(_), do: :page_of_channel_messages
end

Secondly, define your calculation as a module calculation and use it like so:

calculate :channel_messages, MyApp.PageOfChannelMessages, MyApp.Calculations.ChannelMessages
end

defmodule MyApp.Calculations.ChannelMessages do
  use Ash.Calculation

  def load(_, _, context) do
    limit = context[:limit] || 100
    offset = context[:offset] || 0
    [messages: Message |> Ash.Query.limit(limit) |> Ash.Query.offset(offset)]
  end

  def calculate(records, _, _) do
    Enum.map(records, fn record -> 
      %{...construct your page here, the limited/offset records will be in `record.messages`}
    end)
  end
end

I haven’t verified the above code, so you may need to adjust accordingly. Hopefully this gets you somewhere at least

I’ve recently done some major refactoring of relationship loading which came with drastic simplification. I think with that simplification it would be reasonable to support pagination of relationships at the core framework level, which AshGraphql could then use to provide pagination/relay connection support at every level.

[FedeMadoery]

Wow, thanks so much for the fast response…

I will give it a try and post the final result if it works or some other question if I get stuck! Thanks so much

[juno]

This works fine, but the results yields to JsonString in the playground if we set it to :map. If we set it to a type :union i.e., ChannelMessageUnion, we get an error indicating that :list or :map is supported.

defmodule MyApp.PageOfChannelMessages do
  use AshGraphql.Type

  use Ash.Type.NewType, 
    subtype_of: :map,
    constraints: [
      fields: [
          count: [
              type: :integer,
              allow_nil?: false
          ],
          has_next_page: [
              type: :boolean,
              allow_nil?: false
          ],
          results: [
              type: {:array, MyApp.ChannelMessageUnion},
              allow_nil?: false
          ],
          ... Other fields
       ]
    ]

  def graphql_type(_), do: :page_of_channel_messages
end

Trying to see if this is possible todo before we pivot to a different solution.
Thanks.

[zachdaniel]

What’s the full error that you get?

[juno]

This is the compilation error when the results type is set to {:array, MyApp.ChannelMessageUnion}

== Compilation error in file lib/myapp_web/schema.ex ==
** (BadMapError) expected a map, got: nil
    (elixir 1.15.7) lib/map.ex:535: Map.get(nil, :constraints, nil)
    (ash_graphql 0.26.8) lib/resource/resource.ex:3926: AshGraphql.Resource.do_field_type/4
    (ash_graphql 0.26.8) lib/resource/resource.ex:3854: AshGraphql.Resource.do_field_type/4
    (ash_graphql 0.26.8) lib/resource/resource.ex:2673: anonymous fn/5 in AshGraphql.Resource.define_map_types/4
    (elixir 1.15.7) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ash_graphql 0.26.8) lib/resource/resource.ex:2620: anonymous fn/4 in AshGraphql.Resource.define_map_types/4
    (elixir 1.15.7) lib/enum.ex:4317: Enum.flat_map_list/2
    (ash_graphql 0.26.8) lib/resource/resource.ex:2601: anonymous fn/4 in AshGraphql.Resource.map_definitions/3
    (elixir 1.15.7) lib/enum.ex:4317: Enum.flat_map_list/2
    (ash_graphql 0.26.8) lib/resource/resource.ex:1507: AshGraphql.Resource.type_definitions/3
    (ash_graphql 0.26.8) lib/api/api.ex:96: anonymous fn/3 in AshGraphql.Api.type_definitions/6
    (elixir 1.15.7) lib/enum.ex:4317: Enum.flat_map_list/2
    (elixir 1.15.7) lib/enum.ex:4318: Enum.flat_map_list/2
    (ash_graphql 0.26.8) lib/api/api.ex:93: AshGraphql.Api.type_definitions/6
    lib/ohmd_web/schema.ex:6: MyApp.Channels.AshTypes.run/2
    (absinthe 1.7.6) lib/absinthe/pipeline.ex:408: Absinthe.Pipeline.run_phase/3

This is message union implementation

defmodule MyApp.ChannelMessageUnion do
  @moduledoc false
  
  alias MyApp.{ImageChannelMessage, TextChannelMessage}

  @types [
    image: [
      type: ImageChannelMessage,
      tag: :type,
      tag_value: :image_channel_message
    ],
    text: [
      type: TextChannelMessage,
      tag: :type,
      tag_value: :text_channel_message
    ]
  ]

  @structs_to_names Keyword.new(@types, fn {key, _value} ->
                      {key, key}
                    end)

  use Ash.Type.NewType,
    subtype_of: :union,
    constraints: [
      types: @types
    ]

  def struct_to_name(%_struct{} = s), do: @structs_to_names[s.type]

  def graphql_type(_), do: :messages

  def graphql_unnested_unions(_), do: Keyword.keys(@types)
end

[zachdaniel]

Can you try ash_graphql main? I’ve added something to address this.

[juno]

Wow!! this is amazing. Thank you so much.

I just did a build against the main branch and now the initial error has gone away but the Absinthe complains

== Compilation error in file lib/ohmd_web/schema.ex ==
** (Absinthe.Schema.Error) Compilation failed:
---------------------------------------
## Locations
/path/myapp/deps/ash_graphql/lib/resource/resource.ex:1585

:messages is not a valid input type for argument :results because
:messages is an "UnionTypeDefinition". Arguments may only be input types.

Only input types may be used as inputs. Input types may not be used as output types

Input types consist of Scalars, Enums, and Input Objects.
---------------------------------------
## Locations
/path/myapp/deps/ash_graphql/lib/resource/resource.ex:1585

:messages is not a valid type for field :results because
:messages is an UnionTypeDefinition, and this field is part of an InputObjectTypeDefinition.

Only input types may be used as inputs. Input types may not be used as output types

Input types consist of Scalars, Enums, and Input Objects.


    (absinthe 1.7.6) lib/absinthe/schema.ex:392: Absinthe.Schema.__after_compile__/2
    (stdlib 5.1.1) lists.erl:1594: :lists.foldl/3

[zachdaniel]

Ah, right…should be a simple fix, will take a look this evening or tomorrow when I get a chance.

[juno]

Sounds good!

I’ve been looking at the ash_graphql code trying to understand what’s happening. I was able to get it to work by bypassing the generation on the input type within map_definitions function in ash_graphql/lib/resource/resource.ex:2667.

input_type_name =
  if constraints[:fields] do
    if Ash.Type.NewType.new_type?(attribute.type) do
      cond do
        function_exported?(attribute.type, :graphql_input_type, 0) ->
          attribute.type.graphql_input_type()
  
        function_exported?(attribute.type, :graphql_input_type, 1) ->
          attribute.type.graphql_input_type(attribute.constraints)
  
        true ->
          # Here it's generating input_type for the NewType. 
          # Returning nil made it work.

          #map_type(resource, attribute.name, _input? = true)
          nil
      end
    else
      map_type(resource, attribute.name, _input? = true)
    end
  else
    nil
  end

Thanks

[juno]

This got me as far as compiling and querying just fine but when I set the :offset and :limit in the calculation as such.

{MyApp.Calculations.ChannelMessages, :offset, :limit}

It yields, the following

## Locations
/myapp/deps/ash_graphql/lib/resource/resource.ex:2161

In field Eq, :channel_channel_messages_input is not defined in your schema.

Types must exist if referenced.
---------------------------------------
## Locations
/myapp/deps/ash_graphql/lib/resource/resource.ex:2161

In field Not_eq, :channel_channel_messages_input is not defined in your schema.

...More similar errors

Well at this point, I should just wait. Let us know. Thank you for all your help.

[zachdaniel]

I should really reproduce this in a test. However, I believe I’ve isolated the fix. Please try out ash_graphql main.

[juno]

Wow. it’s working now. Awesome!

Looks like, in the module-based calculation, the return type must be a list of items. We got it to work with an inline calculation anonymous function. I see that there is a ticket for this specific issue in ash git repo Anonymous function calculations are inconsistent with module calculations · Issue #625 · ash-project/ash · GitHub

    calculate :channel_messages, MyApp.PageOfChannelMessages, fn record,
                                                                         %{
                                                                           offset: offset,
                                                                           limit: limit
                                                                         } = context ->

   ...Do your thing here

   {:ok, %{Page Of Channel Message info}
end do
      argument :offset, :integer do
        default 0
      end

      argument :limit, :integer do
        default 10
      end
end

I can’t thank you enough for helping us out.

[juno]

One final question. There is little difference b/w the result of the calculation with custom type and regular pagination from the action. the __typename is missing from the graphql query result. Is this something we can control via config or ash_graphql generates?

{
  channel(id: "uuid") {
    channelMessages(offset: 0, limit: 10) {  		
      __typename <----- This is missing from the query response
      count
      hasNextPage
      results {
        ...on TextChannelMessage {
           __typename
          id
        }
        ... more types
      }
    }
  }
}

[zachdaniel]

I would still suggest using a module based calculation. The only thing you need to do is change your module based calculation to map over each record and run your same function. Calculation modules are defined that way so that data can be loaded efficiently.

[zachdaniel]

As for __typename being missing, that is pretty baffling. That isn’t something that we do, that is something that should happen in the underlying absinthe tooling AFAIK.

[juno]

Ok let me give that a shot again

[juno]

The calculation module works and also __typename is returning from the query.

[juno]

My apologies. I spoke too soon. We currently have an issue where the inner results of the page of the channel messages ({:array, Union type}) yields [{}] with 1 message in the channel.

{
  "data": {
    "channel": {
      "channelMessages": {
        "count": 1,
        "hasNextPage": false,
        "results": [{}]
      },
      "id": "74ea615e-ca90-48e2-a422-eeb233d05f0d",
    }
  }
}

If I try the ChannelMessageUnion directly w/o the custom type then that works as intended.

[zachdaniel]

strange. What query did you use to get that response?