How recursively fetch a relationship to generate a non table field in a resource?

Ash Forum Ash Questions / Help
1

Hi,

Say I have the following resource which has a field :parent which references the same resource (allow_nil = true). What I want to generate is a field called path which recursively follows each parent and returns a list of each resource’s slug until parent is empty. This represents a hierarchy:

e.g
id =1, name=Sport, slug=sport,parent_id=null
id=2, name=Local Sport, slug=local-sport,parent_id=1
id=3, name=Soccer, slug=soccer,parent_id=2

For each resource I want to generate a path field such that
id =1, name=Sport, slug=sport,parent_id=null, path=[‘sport’]
id=2, name=Local Sport, slug=local-sport,parent_id=1, path=[‘sport’, ‘local-sport’]
id=3, name=Soccer, slug=soccer,parent_id=2, path=[‘sport’, ‘local-sport’, ‘soccer’]

Could someone assist me in heading in the right direction? I’m looking at calculations vs aggregates and im not too sure which of these are the right choice for this. Furthermore I’m not very certain on how to define such a recursive query

defmodule ContentCove.Articles.Topic do
  use Ash.Resource,
    otp_app: :content_cove,
    domain: ContentCove.Articles,
    data_layer: AshPostgres.DataLayer,
    extensions: [AshAdmin.Resource]

  postgres do
    table "topics"
    repo(ContentCove.Repo)
  end

  admin do
    table_columns [
      :name,
      :slug,
      :path,
      :score,
      :parent_id,
      :inserted_at,
      :updated_at,
      :created_by_id,
      :updated_by_id
    ]
  end

  actions do
    defaults [:read, :destroy]
    default_accept [:name, :score, :parent_id]

    create :create do
      change {ContentCove.Changes.Slugify, source: :name, target: :slug}
      change relate_actor(:created_by)
      change relate_actor(:updated_by)
    end
  end

  attributes do
    uuid_primary_key :id

    attribute :name, :string do
      allow_nil? false
    end

    attribute :slug, :ci_string do
      allow_nil? false
      writable? false
      constraints casing: :lower
    end

    attribute :score, :float do
      default 0.0
    end

    create_timestamp :inserted_at
    update_timestamp :updated_at
  end

  relationships do
    belongs_to :created_by, ContentCove.Accounts.User, allow_nil?: false

    belongs_to :updated_by, ContentCove.Accounts.User, allow_nil?: false

    belongs_to :parent, ContentCove.Articles.Topic, allow_nil?: true
  end

  calculations do
    calculate :path,
              {:array, :ci_string},
              {ContentCove.Calculations.Path, keys: [parent: [:slug, :name]]}
  end

  identities do
    identity :unique_name, [:name]
  end
end

Thanks

2

Hey there!

You have a few options here, but most likely your best bet is going to be to leverage your postgres database

ltree (postgres feature)

lree requires some additional work whenever changing data to ensure each element’s path is correct, but it can make working with tree structures much easier.

With ltree, you can define relationships that fetch all ancestors:

has_many :ancestors, __MODULE__ do
  no_attributes? true
  filter expr(fragment("? @> ?", path, parent(path))
end

Then your calculation can express a dependency on :ancestors, i.e

def load(_, _, _) do
  [:ancestors]
end

Then to actually calculate it you can use some simple-ish code to figure out which order they go in based on their parent ids.

recursive CTEs, either in a view or a manual relationship (postgres feature)

With recursive CTEs, you can do a couple different things. You can define a postgres view that produces the structure you want. Or you could use a joinable manual relationship. This is not the simplest strategy, but is effective in a pinch :slight_smile:

https://hexdocs.pm/ash_postgres/manual-relationships.html

Calculate it manually (potentially easier, slower, perhaps a good prototype)

defmodule ContentCove.Calculations.Path do
  use Ash.Resource.Calculation

  require Ash.Query

  def calculate(topics, _, _) do
    topic_map = fetch_all_topics(topics)
    
    Enum.map(topics, fn topic -> 
      # build the path from the fetched topics
    end)
  end

  def fetch_all_topics(topics, acc \\ %{}) do
    acc = Map.merge(acc, Map.new(topics, fn topic -> {topic.id, topic} end))

    topics
    |> Enum.map(&(&1.parent_id))
    |> Enum.filter(fn parent_id -> 
      parent_id && !Map.has_key?(acc, parent_id)
    end)
    |> case do
      [] -> acc # nothing left to fetch, return the map
      unfetched_parent_ids ->
        Topic
        |> Ash.Query.filter(id in ^unfetched_parent_ids)
        |> Ash.read!()
        |> fetch_all_topics(acc)
    end
  end
end

There are 100% some errors in that code, I haven’t run it myself I just typed it up in this window :slight_smile: The important things to note there are that you can run queries in calculate/3, when necessary, and that you should make sure to run those queries based on the list of records, as opposed to each individual record whenever possible.

Doing the work based on a list of records makes the calculate/3 approach actually doable, as it will only make as many queries as your deepest level of nesting, instead of one query per topic in the structure.

2 Likes
3

Cheers Zach!

So I have it working I think with the following:

defmodule ContentCove.Calculations.Path do
  use Ash.Resource.Calculation

  alias ContentCove.Articles.Topic

  require Ash.Query

  @impl true
  def calculate(topic, _, _) do
    fetch_all_topics(topic)
    |> Enum.map(fn {_, topic} -> to_string(topic.slug) end)
    |> Enum.reverse()
    |> IO.inspect()

    # require IEx
    # IEx.pry()
  end

  defp fetch_all_topics(topics, acc \\ %{}) do
    acc = Map.merge(acc, Map.new(topics, fn topic -> {topic.id, topic} end))

    topics
    |> Enum.map(& &1.parent_id)
    |> Enum.filter(fn parent_id ->
      parent_id && !Map.has_key?(acc, parent_id)
    end)
    |> case do
      # nothing left to fetch, return the map
      [] ->
        acc

      unfetched_parent_ids ->
        Topic
        |> Ash.Query.filter(id in ^unfetched_parent_ids)
        |> Ash.read!()
        |> fetch_all_topics(acc)
    end
  end
end

For now I’ve gone with prototype way since the depth of the tree wont really go past 3/4 but I’ll look into ltree / CTE at some point. I’d like to first understand how this works so I can begin piecing things together slowly

However I cant actually confirm 100% outside of using IEX and loading the path. How can I get the :path calculation field to show in the admin? Either in the objects detail page or in the read list page.

Furthermore, how do you go about debugging? I come from a python background and usually put pdb breakpoints and then step into and observe what is happening. Is it easiest to use iex -S mix phx.server with IEx.pry() or do you have a recommendation on how best to debug specifically in the context of Phoenix / Ash?

I found for example errors in my calculation wouldn’t show unless I invoked it specifically via the shell but maybe that’s because it isn’t being loaded via the admin.

4

You can show it in the table by setting table_columns [...., :your_calculation].

For debugging, I’ll typically use IEx.pry(), or dbg/1/IO.inspect/1 in the calculation.

I don’t think there is currently a way to load calculations on the show page of a record. PRs/improvement requests welcome on that front :bowing_man:

5

Its odd with the following in my Resource

  calculations do
    calculate :path,
              {:array, :string},
              ContentCove.Calculations.Path
  end
  admin do
    table_columns [
      :path,
      :slug,
      :name,
      :score,
      :inserted_at,
      :updated_at,
      :created_by_id,
      :updated_by_id
    ]
  end

and the calculation module:

defmodule ContentCove.Calculations.Path do
  use Ash.Resource.Calculation

  alias ContentCove.Articles.Topic

  require Ash.Query
  require IEx

  @impl true
  def calculate(topic, _, _) do
    dbg(topic)
    IO.inspect(~c">>>>>>>>>>>>>")
    IO.inspect(topic)

    fetch_all_topics(topic)
    |> Enum.map(fn {_, topic} -> to_string(topic.slug) end)
    |> Enum.reverse()

    # |> IO.inspect()

    # require IEx
    # IEx.pry()
  end

  defp fetch_all_topics(topics, acc \\ %{}) do
    acc = Map.merge(acc, Map.new(topics, fn topic -> {topic.id, topic} end))

    topics
    |> Enum.map(& &1.parent_id)
    |> Enum.filter(fn parent_id ->
      parent_id && !Map.has_key?(acc, parent_id)
    end)
    |> case do
      # nothing left to fetch, return the map
      [] ->
        acc

      unfetched_parent_ids ->
        Topic
        |> Ash.Query.filter(id in ^unfetched_parent_ids)
        |> Ash.read!()
        |> fetch_all_topics(acc)
    end
  end
end

I cant get the path to show on the admin read page. The column for path is there but its always blank and the Calculation module code is never run. Any ideas?

6

are you on the latest ash_admin version?

7

Oh right, ty! Updated and had to add to add this to the calculation to stop errors:

  @impl true
  def load(_query, _opts, _context) do
    [:parent]
  end

on the resource :path is defined as

  calculations do
    calculate :path,
              {:array, :string},
              ContentCove.Calculations.Path
  end

However the values displayed dont seem correct since path is supposed to return an array of strings. In the admin ui i see single string values that are seemingly incorrect for each topic.

image
image1200×467 43.9 KB

defmodule ContentCove.Calculations.Path do
  use Ash.Resource.Calculation

  alias ContentCove.Articles.Topic

  require Ash.Query

  @impl true
  def calculate(topic, _, _) do
    fetch_all_topics(topic)
    |> Enum.map(fn {_, topic} -> to_string(topic.slug) end)
    |> IO.inspect()
    |> Enum.reverse()
  end

  @impl true
  def load(_query, _opts, _context) do
    [:parent]
  end

  defp fetch_all_topics(topics, acc \\ %{}) do
    acc = Map.merge(acc, Map.new(topics, fn topic -> {topic.id, topic} end))

    #dbg(acc)

    topics
    |> Enum.map(& &1.parent_id)
    |> Enum.filter(fn parent_id ->
      parent_id && !Map.has_key?(acc, parent_id)
    end)
    |> case do
      [] ->
        acc

      unfetched_parent_ids ->
        Topic
        |> Ash.Query.filter(id in ^unfetched_parent_ids)
        |> Ash.read!()
        |> fetch_all_topics(acc)
    end
  end
end

Judging by the terminal output it looks like a single array is built and somehow each row in the admin ui is being given 1 value from the array i.e ["entertainment", "local-sport", "celebrities", "sport", "cricket"] that is the output from the inspect above in calculate.

If i hardcode ["test1", "test2"] to be returned from the calculate function in the admin ui it shows 1 value from the array for each row and only returns 2 rows instead of the correct number shown in the screenshot. Furthermore I though maybe its supposed to return an array of arrays for each row but that too only displays s single value in the admin for each row :thinking:

8

Maybe I spoke to soon if I return an array of arrays and it matches the number of rows then it looks like path just includes all those strings.

E.g [["test1", "test1-sub1", "test1-sub2"], ["test2"], ["test3"], ["test4"], ["test5"]]

image
image1200×471 43.7 KB

Long shot but is there a way to change the formatting in the admin ui for how array’s are displayed? Atm array of strings for example in the screenshot above aren’t comma delimited (or formatted as a list in any way) which makes readability slightly difficult

Looks like I need to update my calculate function to do something similar. I was under the impression that it would run for each row in the resource separately? Still digging into it so maybe I’ve jumped the gun once again

9

Calculations run on batches of records. So in your case you’d map over each record and return a value for each one.

10

I’d suggest writing a calculation that depends on the list calculation and turns it into a CSV value for displaying in the admin.