Ash is being extremelly slow to load a relationship of my resource

Ash Forum Ash Questions / Help
,
1

I have the following resource in my system:

defmodule Pacman.Markets.Record do
  @moduledoc false

  use Ash.Resource,
    data_layer: AshPostgres.DataLayer,
    extensions: [UUIDV7]

  attributes do
    uuid_v7_primary_key :id, writable?: true

    timestamps(private?: false)
  end

  relationships do
    alias Pacman.Markets.Entity

    belongs_to :grantee_entity, Entity do
      allow_nil? true
      attribute_writable? true

      destination_attribute :id
    end
  end

  postgres do
    table "records"

    repo Pacman.Repo
  end
end

In my Entityresource, I have the following action:

read :all_stale do
      filter expr(status == :stale)

      prepare build(limit: 1000)
    end

And relationship:

    has_many :grantee_records, Record do
      destination_attribute :grantee_entity_id
    end

Now I’m trying to run that action and load the grantee_records relationship:

iex(1)> Pacman.Markets.Entity.all_stale() |> Pacman.Markets.load!(:grantee_records)
[debug] QUERY OK source="entities" db=6.9ms queue=0.7ms idle=1283.2ms
SELECT e0."id", e0."first_name", e0."middle_name", e0."last_name", e0."name_suffix", e0."full_name", e0."address_house_number", e0."address_street_direction", e0."address_street_name", e0."address_street_suffix", e0."address_street_post_direction", e0."address_unit_prefix", e0."address_unit_value", e0."address_city", e0."address_state", e0."address_zip", e0."address_zip_4", e0."address_legacy", e0."address_normalized", e0."address", e0."status", e0."total_records", e0."total_buy_records", e0."total_sell_records", e0."fix_and_flip?", e0."buy_and_hold?", e0."wholesaler?", e0."year_before_last", e0."year_before_last_buy_records", e0."year_before_last_buy_transfer_amount", e0."year_before_last_sell_records", e0."year_before_last_sell_transfer_amount", e0."last_year", e0."last_year_buy_records", e0."last_year_buy_transfer_amount", e0."last_year_sell_records", e0."last_year_sell_transfer_amount", e0."current_year", e0."current_year_buy_records", e0."current_year_buy_transfer_amount", e0."current_year_sell_records", e0."current_year_sell_transfer_amount", e0."score", e0."score_version", e0."inserted_at", e0."updated_at" FROM "entities" AS e0 WHERE (e0."status"::varchar = $1::varchar) LIMIT $2 [:stale, 1000]
[debug] QUERY OK source="records" db=44.2ms decode=0.1ms queue=4.2ms idle=1441.2ms
SELECT r0."id", r0."inserted_at", r0."updated_at", r0."grantee_entity_id", r0."grantor_entity_id" FROM "records" AS r0 WHERE (r0."grantee_entity_id"::uuid IN ($1::uuid,$2::uuid,$3::uuid,$4::uuid,$5::uuid,$6::uuid,$7::uuid,$8::uuid,$9::uuid,$10::uuid,$11::uuid,$12::uuid,$13::uuid,$14::uuid,$15::uuid,$16::uuid,$17::uuid,$18::uuid,$19::uuid,$20::uuid,$21::uuid,$22::uuid,$23::uuid,$24::uuid,$25::uuid,$26::uuid,$27::uuid,$28::uuid,$29::uuid,$30::uuid,$31::uuid,$32::uuid,$33::uuid,$34::uuid,$35::uuid,$36::uuid,$37::uuid,$38::uuid,$39::uuid,$40::uuid,$41::uuid,$42::uuid,$43::uuid,$44::uuid,$45::uuid,$46::uuid,$47::uuid,$48::uuid,$49::uuid,$50::uuid,$51::uuid,$52::uuid,$53::uuid,$54::uuid,$55::uuid,$56::uuid,$57::uuid,$58::uuid,$59::uuid,$60::uuid,$61::uuid,$62::uuid,$63::uuid,$64::uuid,$65::uuid,$66::uuid,$67::uuid,$68::uuid,$69::uuid,$70::uuid,$71::uuid,$72::uuid,$73::uuid,$74::uuid,$75::uuid,$76::uuid,$77::uuid,$78::uuid,$79::uuid,$80::uuid,$81::uuid,$82::uuid,$83::uuid,$84::uuid,$85::uuid,$86::uuid,$87::uuid,$88::uuid,$89::uuid,$90::uuid,$91::uuid,$92::uuid,$93::uuid,$94::uuid,$95::uuid,$96::uuid,$97::uuid,$98::uuid,$99::uuid,$100::uuid,$101::uuid,$102::uuid,$103::uuid,$104::uuid,$105::uuid,$106::uuid,$107::uuid,$108::uuid,$109::uuid,$110::uuid,$111::uuid,$112::uuid,$113::uuid,$114::uuid,$115::uuid,$116::uuid,$117::uuid,$118::uuid,$119::uuid,$120::uuid,$121::uuid,$122::uuid,$123::uuid,$124::uuid,$125::uuid,$126::uuid,$127::uuid,$128::uuid,$129::uuid,$130::uuid,$131::uuid,$132::uuid,$133::uuid,$134::uuid,$135::uuid,$136::uuid,$137::uuid,$138::uuid,$139::uuid,$140::uuid,$141::uuid,$142::uuid,$143::uuid,$144::uuid,$145::uuid,$146::uuid,$147::uuid,$148::uuid,$149::uuid,$150::uuid,$151::uuid,$152::uuid,$153::uuid,$154::uuid,$155::uuid,$156::uuid,$157::uuid,$158::uuid,$159::uuid,$160::uuid,$161::uuid,$162::uuid,$163::uuid,$164::uuid,$165::uuid,$166::uuid,$167::uuid,$168::uuid,$169::uuid,$170::uuid,$171::uuid,$172::uuid,$173::uuid,$174::uuid,$175::uuid,$176::uuid,$177::uuid,$178::uuid,$179::uuid,$180::uuid,$181::uuid,$182::uuid,$183::uuid,$184::uuid,$185::uuid,$186::uuid,$187::uuid,$188::uuid,$189::uuid,$190::uuid,$191::uuid,$192::uuid,$193::uuid,$194::uuid,$195::uuid,$196::uuid,$197::uuid,$198::uuid,$199::uuid,$200::uuid,$201::uuid,$202::uuid,$203::uuid,$204::uuid,$205::uuid,$206::uuid,$207::uuid,$208::uuid,$209::uuid,$210::uuid,$211::uuid,$212::uuid,$213::uuid,$214::uuid,$215::uuid,$216::uuid,$217::uuid,$218::uuid,$219::uuid,$220::uuid,$221::uuid,$222::uuid,$223::uuid,$224::uuid,$225::uuid,$226::uuid,$227::uuid,$228::uuid,$229::uuid,$230::uuid,$231::uuid,$232::uuid,$233::uuid,$234::uuid,$235::uuid,$236::uuid,$237::uuid,$238::uuid,$239::uuid,$240::uuid,$241::uuid,$242::uuid,$243::uuid,$244::uuid,$245::uuid,$246::uuid,$247::uuid,$248::uuid,$249::uuid,$250::uuid,$251::uuid,$252::uuid,$253::uuid,$254::uuid,$255::uuid,$256::uuid,$257::uuid,$258::uuid,$259::uuid,$260::uuid,$261::uuid,$262::uuid,$263::uuid,$264::uuid,$265::uuid,$266::uuid,$267::uuid,$268::uuid,$269::uuid,$270::uuid,$271::uuid,$272::uuid,$273::uuid,$274::uuid,$275::uuid,$276::uuid,$277::uuid,$278::uuid,$279::uuid,$280::uuid,$281::uuid,$282::uuid,$283::uuid,$284::uuid,$285::uuid,$286::uuid,$287::uuid,$288::uuid,$289::uuid,$290::uuid,$291::uuid,$292::uuid,$293::uuid,$294::uuid,$295::uuid,$296::uuid,$297::uuid,$298::uuid,$299::uuid,$300::uuid,$301::uuid,$302::uuid,$303::uuid,$304::uuid,$305::uuid,$306::uuid,$307::uuid,$308::uuid,$309::uuid,$310::uuid,$311::uuid,$312::uuid,$313::uuid,$314::uuid,$315::uuid,$316::uuid,$317::uuid,$318::uuid,$319::uuid,$320::uuid,$321::uuid,$322::uuid,$323::uuid,$324::uuid,$325::uuid,$326::uuid,$327::uuid,$328::uuid,$329::uuid,$330::uuid,$331::uuid,$332::uuid,$333::uuid,$334::uuid,$335::uuid,$336::uuid,$337::uuid,$338::uuid,$339::uuid,$340::uuid,$341::uuid,$342::uuid,$343::uuid,$344::uuid,$345::uuid,$346::uuid,$347::uuid,$348::uuid,$349::uuid,$350::uuid,$351::uuid,$352::uuid,$353::uuid,$354::uuid,$355::uuid,$356::uuid,$357::uuid,$358::uuid,$359::uuid,$360::uuid,$361::uuid,$362::uuid,$363::uuid,$364::uuid,$365::uuid,$366::uuid,$367::uuid,$368::uuid,$369::uuid,$370::uuid,$371::uuid,$372::uuid,$373::uuid,$374::uuid,$375::uuid,$376::uuid,$377::uuid,$378::uuid,$379::uuid,$380::uuid,$381::uuid,$382::uuid,$383::uuid,$384::uuid,$385::uuid,$386::uuid,$387::uuid,$388::uuid,$389::uuid,$390::uuid,$391::uuid,$392::uuid,$393::uuid,$394::uuid,$395::uuid,$396::uuid,$397::uuid,$398::uuid,$399::uuid,$400::uuid,$401::uuid,$402::uuid,$403::uuid,$404::uuid,$405::uuid,$406::uuid,$407::uuid,$408::uuid,$409::uuid,$410::uuid,$411::uuid,$412::uuid,$413::uuid,$414::uuid,$415::uuid,$416::uuid,$417::uuid,$418::uuid,$419::uuid,$420::uuid,$421::uuid,$422::uuid,$423::uuid,$424::uuid,$425::uuid,$426::uuid,$427::uuid,$428::uuid,$429::uuid,$430::uuid,$431::uuid,$432::uuid,$433::uuid,$434::uuid,$435::uuid,$436::uuid,$437::uuid,$438::uuid,$439::uuid,$440::uuid,$441::uuid,$442::uuid,$443::uuid,$444::uuid,$445::uuid,$446::uuid,$447::uuid,$448::uuid,$449::uuid,$450::uuid,$451::uuid,$452::uuid,$453::uuid,$454::uuid,$455::uuid,$456::uuid,$457::uuid,$458::uuid,$459::uuid,$460::uuid,$461::uuid,$462::uuid,$463::uuid,$464::uuid,$465::uuid,$466::uuid,$467::uuid,$468::uuid,$469::uuid,$470::uuid,$471::uuid,$472::uuid,$473::uuid,$474::uuid,$475::uuid,$476::uuid,$477::uuid,$478::uuid,$479::uuid,$480::uuid,$481::uuid,$482::uuid,$483::uuid,$484::uuid,$485::uuid,$486::uuid,$487::uuid,$488::uuid,$489::uuid,$490::uuid,$491::uuid,$492::uuid,$493::uuid,$494::uuid,$495::uuid,$496::uuid,$497::uuid,$498::uuid,$499::uuid,$500::uuid,$501::uuid,$502::uuid,$503::uuid,$504::uuid,$505::uuid,$506::uuid,$507::uuid,$508::uuid,$509::uuid,$510::uuid,$511::uuid,$512::uuid,$513::uuid,$514::uuid,$515::uuid,$516::uuid,$517::uuid,$518::uuid,$519::uuid,$520::uuid,$521::uuid,$522::uuid,$523::uuid,$524::uuid,$525::uuid,$526::uuid,$527::uuid,$528::uuid,$529::uuid,$530::uuid,$531::uuid,$532::uuid,$533::uuid,$534::uuid,$535::uuid,$536::uuid,$537::uuid,$538::uuid,$539::uuid,$540::uuid,$541::uuid,$542::uuid,$543::uuid,$544::uuid,$545::uuid,$546::uuid,$547::uuid,$548::uuid,$549::uuid,$550::uuid,$551::uuid,$552::uuid,$553::uuid,$554::uuid,$555::uuid,$556::uuid,$557::uuid,$558::uuid,$559::uuid,$560::uuid,$561::uuid,$562::uuid,$563::uuid,$564::uuid,$565::uuid,$566::uuid,$567::uuid,$568::uuid,$569::uuid,$570::uuid,$571::uuid,$572::uuid,$573::uuid,$574::uuid,$575::uuid,$576::uuid,$577::uuid,$578::uuid,$579::uuid,$580::uuid,$581::uuid,$582::uuid,$583::uuid,$584::uuid,$585::uuid,$586::uuid,$587::uuid,$588::uuid,$589::uuid,$590::uuid,$591::uuid,$592::uuid,$593::uuid,$594::uuid,$595::uuid,$596::uuid,$597::uuid,$598::uuid,$599::uuid,$600::uuid,$601::uuid,$602::uuid,$603::uuid,$604::uuid,$605::uuid,$606::uuid,$607::uuid,$608::uuid,$609::uuid,$610::uuid,$611::uuid,$612::uuid,$613::uuid,$614::uuid,$615::uuid,$616::uuid,$617::uuid,$618::uuid,$619::uuid,$620::uuid,$621::uuid,$622::uuid,$623::uuid,$624::uuid,$625::uuid,$626::uuid,$627::uuid,$628::uuid,$629::uuid,$630::uuid,$631::uuid,$632::uuid,$633::uuid,$634::uuid,$635::uuid,$636::uuid,$637::uuid,$638::uuid,$639::uuid,$640::uuid,$641::uuid,$642::uuid,$643::uuid,$644::uuid,$645::uuid,$646::uuid,$647::uuid,$648::uuid,$649::uuid,$650::uuid,$651::uuid,$652::uuid,$653::uuid,$654::uuid,$655::uuid,$656::uuid,$657::uuid,$658::uuid,$659::uuid,$660::uuid,$661::uuid,$662::uuid,$663::uuid,$664::uuid,$665::uuid,$666::uuid,$667::uuid,$668::uuid,$669::uuid,$670::uuid,$671::uuid,$672::uuid,$673::uuid,$674::uuid,$675::uuid,$676::uuid,$677::uuid,$678::uuid,$679::uuid,$680::uuid,$681::uuid,$682::uuid,$683::uuid,$684::uuid,$685::uuid,$686::uuid,$687::uuid,$688::uuid,$689::uuid,$690::uuid,$691::uuid,$692::uuid,$693::uuid,$694::uuid,$695::uuid,$696::uuid,$697::uuid,$698::uuid,$699::uuid,$700::uuid,$701::uuid,$702::uuid,$703::uuid,$704::uuid,$705::uuid,$706::uuid,$707::uuid,$708::uuid,$709::uuid,$710::uuid,$711::uuid,$712::uuid,$713::uuid,$714::uuid,$715::uuid,$716::uuid,$717::uuid,$718::uuid,$719::uuid,$720::uuid,$721::uuid,$722::uuid,$723::uuid,$724::uuid,$725::uuid,$726::uuid,$727::uuid,$728::uuid,$729::uuid,$730::uuid,$731::uuid,$732::uuid,$733::uuid,$734::uuid,$735::uuid,$736::uuid,$737::uuid,$738::uuid,$739::uuid,$740::uuid,$741::uuid,$742::uuid,$743::uuid,$744::uuid,$745::uuid,$746::uuid,$747::uuid,$748::uuid,$749::uuid,$750::uuid,$751::uuid,$752::uuid,$753::uuid,$754::uuid,$755::uuid,$756::uuid,$757::uuid,$758::uuid,$759::uuid,$760::uuid,$761::uuid,$762::uuid,$763::uuid,$764::uuid,$765::uuid,$766::uuid,$767::uuid,$768::uuid,$769::uuid,$770::uuid,$771::uuid,$772::uuid,$773::uuid,$774::uuid,$775::uuid,$776::uuid,$777::uuid,$778::uuid,$779::uuid,$780::uuid,$781::uuid,$782::uuid,$783::uuid,$784::uuid,$785::uuid,$786::uuid,$787::uuid,$788::uuid,$789::uuid,$790::uuid,$791::uuid,$792::uuid,$793::uuid,$794::uuid,$795::uuid,$796::uuid,$797::uuid,$798::uuid,$799::uuid,$800::uuid,$801::uuid,$802::uuid,$803::uuid,$804::uuid,$805::uuid,$806::uuid,$807::uuid,$808::uuid,$809::uuid,$810::uuid,$811::uuid,$812::uuid,$813::uuid,$814::uuid,$815::uuid,$816::uuid,$817::uuid,$818::uuid,$819::uuid,$820::uuid,$821::uuid,$822::uuid,$823::uuid,$824::uuid,$825::uuid,$826::uuid,$827::uuid,$828::uuid,$829::uuid,$830::uuid,$831::uuid,$832::uuid,$833::uuid,$834::uuid,$835::uuid,$836::uuid,$837::uuid,$838::uuid,$839::uuid,$840::uuid,$841::uuid,$842::uuid,$843::uuid,$844::uuid,$845::uuid,$846::uuid,$847::uuid,$848::uuid,$849::uuid,$850::uuid,$851::uuid,$852::uuid,$853::uuid,$854::uuid,$855::uuid,$856::uuid,$857::uuid,$858::uuid,$859::uuid,$860::uuid,$861::uuid,$862::uuid,$863::uuid,$864::uuid,$865::uuid,$866::uuid,$867::uuid,$868::uuid,$869::uuid,$870::uuid,$871::uuid,$872::uuid,$873::uuid,$874::uuid,$875::uuid,$876::uuid,$877::uuid,$878::uuid,$879::uuid,$880::uuid,$881::uuid,$882::uuid,$883::uuid,$884::uuid,$885::uuid,$886::uuid,$887::uuid,$888::uuid,$889::uuid,$890::uuid,$891::uuid,$892::uuid,$893::uuid,$894::uuid,$895::uuid,$896::uuid,$897::uuid,$898::uuid,$899::uuid,$900::uuid,$901::uuid,$902::uuid,$903::uuid,$904::uuid,$905::uuid,$906::uuid,$907::uuid,$908::uuid,$909::uuid,$910::uuid,$911::uuid,$912::uuid,$913::uuid,$914::uuid,$915::uuid,$916::uuid,$917::uuid,$918::uuid,$919::uuid,$920::uuid,$921::uuid,$922::uuid,$923::uuid,$924::uuid,$925::uuid,$926::uuid,$927::uuid,$928::uuid,$929::uuid,$930::uuid,$931::uuid,$932::uuid,$933::uuid,$934::uuid,$935::uuid,$936::uuid,$937::uuid,$938::uuid,$939::uuid,$940::uuid,$941::uuid,$942::uuid,$943::uuid,$944::uuid,$945::uuid,$946::uuid,$947::uuid,$948::uuid,$949::uuid,$950::uuid,$951::uuid,$952::uuid,$953::uuid,$954::uuid,$955::uuid,$956::uuid,$957::uuid,$958::uuid,$959::uuid,$960::uuid,$961::uuid,$962::uuid,$963::uuid,$964::uuid,$965::uuid,$966::uuid,$967::uuid,$968::uuid,$969::uuid,$970::uuid,$971::uuid,$972::uuid,$973::uuid,$974::uuid,$975::uuid,$976::uuid,$977::uuid,$978::uuid,$979::uuid,$980::uuid,$981::uuid,$982::uuid,$983::uuid,$984::uuid,$985::uuid,$986::uuid,$987::uuid,$988::uuid,$989::uuid,$990::uuid,$991::uuid,$992::uuid,$993::uuid,$994::uuid,$995::uuid,$996::uuid,$997::uuid,$998::uuid,$999::uuid,$1000::uuid)) ["02vTUOqTJXVX5ldag1zgaD", "02vTUOrgTDgBI2dpDcAkex", "02vTUOuNqUP4yHJHBrKRQJ", "02vTUOpFHbO3m7iYJSvHD0", "02vTUOr3fpn5sVnD7ZVuxH", "02vTUOqSL1tYReP9Sk99qF", "02vTUOrgNIJTeSGETDEQUp", "02vTUOuN485drbsrLshYfv", "02vTUOr4GNSawIHFOUqHSK", "02vTUOtf6R2gQoxNJ11lsY", "02vTUOt60aR2ivCzKfPgy0", "02vTUOux8W69JiXdSs0t2O", "02vTUOrh3jCLI8FMTF6wHv", "02vTUOpFZTuIK6OU3u4Axi", "02vTUOuwq97Y0ylmPpvMg4", "02vTUOqRqpM7ZBsjnKG8LO", "02vTUOsRqoB0Nb9S6XgVGH", "02vTUOprgjv6ao5iT2vEuf", "02vTUOteuTSwOL6r5nnTHD", "02vTUOr4wPnZONpEjQO6TX", "02vTUOqT1ZXpDM1QcMU4Dx", "02vTUOuNYI1OuLilJLmyzh", "02vTUOuwM9wwkHjbnJqQ8U", "02vTUOuMs5HBGVeR5qsMPq", "02vTUOuwqLUL6BJnQSaT5j", "02vTUOoRMzYLl0xQ7yaK7T", "02vTUOteiF5ya7KoM8oDFt", "02vTUOoR56jfHznMytIYL5", "02vTUOt6nB1NXDSHq80g5K", "02vTUOpFraqsWZ3k5WhAwc", "02vTUOrfOqrEblzGuCMkgU", "02vTUOr4434LkAYjfSSZBp", "02vTUOsQNuQlXLo7Ydk2XO", "02vTUOr44HV7zztjrkP2mA", "02vTUOrgxmmNhtctom9iot", "02vTUOsQI7Y62EDmAbHPk1", "02vTUOt5uXn5XARaDoSDJV", "02vTUOuOQcjUv8utHlBPQK", "02vTUOr4SK2eLO3eeSfa9a", "02vTUOqTPa6cPScCNbEdY7", "02vTUOuOQlwsiXi2CPbqoP", "02vTUOuMryQTuYpoki6c0b", "02vTUOr44AxvVVxTdnacU4", "02vTUOtfOgVrNntAmtAgFE", "02vTUOr3xutvssGtS1cphY", "02vTUOqTJUWpmZLkgL1ZYA", "02vTUOrfP0futaRy1e6LwF", "02vTUOr4G8KxBSBAZ914r6", "02vTUOr44HE5u9KHIKFLAn", "02vTUOtfCYkPqEu1jQS2TV", ...]

As you can see from the logs, the action and load queries run pretty fast (6.9ms, 44.2ms), but after the load query runs, it will take around 1~5 minutes for the Pacman.Markets.load! call to return the values, and during that time the erlang VM will be at 100% CPU.

I tested it both the ash and ash_postgres using main and latest version.

2

Can you see how it behaves if you remove the limit option?

3

Hmm, my table has 5 million rows, I can try without the limit, but I think it would be very slow just because of the number of rows.

4

Ah, nvm i misread your example. Can you show me what the primary read action looks like on the destination resource? How many related entities do you expect to see typically?

5

In both resources the primary read is the default one.

This query is returning 1000 entities (because of the limit) and around 5500 records in total

6

That shouldn’t be a significant issue at all. I’m actually completely rewriting relationship and calculation loading currently (not because it was too slow) so perhaps we’ll get lucky and my current efforts will solve this problem. I’ll push up my latest changes for you to try. It’s not ready to be merged quite yet, but will tell us a lot if you don’t have the same issues.

7

What does Ash.Type.simple_equality?(YourUUIDType) return?

8 
iex(52)> Ash.Type.simple_equality?(UUIDV7.Type)
false
9

Ah, okay. Can I see your equality implementation? Is it necessary for it to be overridden? We will only ever call it with properly casted values, so it’s very rare to need to overrride it. I believe that is the source of your issues. If you delete the function from the type you should see a significant improvement I suspect.

10

Here is the type full implementation:

defmodule UUIDV7.Type do
  @moduledoc """
  UUID V7 type
  """

  alias UUIDV7.Encoder

  use Ash.Type

  @impl true
  def storage_type, do: :uuid

  @impl true
  def generator(_constraints) do
    {:ok, term} = Uniq.UUID.uuid7() |> process(:raw, :encoded)

    term
  end

  @impl true
  def cast_input(term, _constraints), do: process(term, identify_format(term), :encoded)

  @impl true
  def cast_stored(term, _constraints), do: process(term, identify_format(term), :encoded)

  @impl true
  def dump_to_native(term, _constraints), do: process(term, identify_format(term), :integer)

  @impl true
  def dump_to_embedded(term, _constraints), do: process(term, identify_format(term), :raw)

  @impl true
  def equal?(lhs, rhs),
    do: process(lhs, identify_format(lhs), :raw) == process(rhs, identify_format(rhs), :raw)

  defp process(term, initial_format, requested_format) do
    term
    |> validate(initial_format)
    |> restore(initial_format, requested_format)
    |> decode(initial_format, requested_format)
    |> encode(requested_format)
    |> dump(initial_format, requested_format)
  end

  defp validate(term, initial_format)

  defp validate(term, initial_format) when initial_format in [:integer, :raw] do
    if Uniq.UUID.valid?(term), do: {:ok, term}, else: {:error, "got invalid term"}
  end

  defp validate(term, :encoded), do: {:ok, term}
  defp validate(nil, _initial_format), do: {:ok, nil}
  defp validate(_term, _initial_format), do: {:error, "got invalid term"}

  defp restore(result, initial_format, requested_format)

  defp restore({:ok, term}, :integer, requested_format) when requested_format in [:raw, :encoded],
    do: {:ok, Uniq.UUID.to_string(term)}

  defp restore({:ok, term}, _initial_format, _requested_format), do: {:ok, term}

  defp decode(result, initial_format, requested_format)
  defp decode({:ok, nil}, nil, _requested_format), do: {:ok, nil}
  defp decode({:ok, term}, :encoded, _requested_format), do: Encoder.decode(term)
  defp decode({:ok, term}, _initial_format, _requested_format), do: {:ok, term}

  defp encode(result, requested_format)
  defp encode({:ok, nil}, _requested_format), do: {:ok, nil}
  defp encode({:ok, term}, :encoded), do: Encoder.encode(term)
  defp encode({:ok, term}, _requested_format), do: {:ok, term}

  defp dump(result, initial_format, requested_format)

  defp dump({:ok, term}, initial_format, :integer) when initial_format in [:raw, :encoded] do
    {:ok, Uniq.UUID.string_to_binary!(term)}
  rescue
    _error in ArgumentError -> {:error, "can not dump term"}
  end

  defp dump({:ok, term}, _initial_format, _requested_format), do: {:ok, term}

  defp identify_format(
         <<_::binary-size(8), ?-, _::binary-size(4), ?-, _::binary-size(4), ?-, _::binary-size(4),
           ?-, _::binary-size(12)>>
       ),
       do: :raw

  defp identify_format(<<_::binary-size(22)>>), do: :encoded
  defp identify_format(<<_::128>>), do: :integer

  defp identify_format(string) when is_binary(string) do
    case String.split(string, "_") do
      [<<_::binary-size(32)>>] -> :hex
      _ -> :unknown
    end
  end

  defp identify_format(nil), do: nil
  defp identify_format(_term), do: :unknown
end

It is basically the implementation from this library GitHub - zoonect-oss/ash_uuid: AshUUID: Extension for using UUID v4 and v7, with supports encoding and prefixing but I removed some of the features.

I think I still need it because the type converts from and to base62, so the equal function is doing that work before comparing them.

11

But yeah, you are correct, removing the equal function make it extremely fast.

Is there some workaround for this? I tried changing the equal function to just be lhs == rhs just to see if it was the process step that was slow, but even that made it super slow.

Otherwise I will just remove the base62 support for now.

12

If it does that conversion when casting/loading from the database, then it should not be necessary to have an equality implementation that does it as well.

13

The basic issue is that if there is a custom equality function, we can’t use maps when connecting records retrieved from the database.

14

That makes sense, I will remove the equal part and see how it goes

15

@zachdaniel I think I found another problem.

If I have my action like this:

    read :all_stale do
      filter expr(status == :stale)

      prepare build(limit: 10000)
    end

And run this code:

Pacman.Markets.Entity.all_stale!() |> Pacman.Markets.load!([:grantee_records, :grantor_records])

It runs in about 8 seconds.

Now, if I change the action to this:

    read :all_stale do
      pagination keyset?: true

      filter expr(status == :stale)
    end

And run the code like this:

  Pacman.Markets.Entity
    |> Ash.Query.new()
    |> Ash.Query.for_read(:all_stale, %{})
    |> Pacman.Markets.stream!(batch_size: 10_000)
    |> Stream.chunk_every(10_000)
    |> Stream.each(fn entities ->
      IO.puts("Got here 1 #{DateTime.utc_now()}")

      entities = Pacman.Markets.load!(entities, [:grantee_records, :grantor_records])

      IO.puts("Got here 2 #{DateTime.utc_now()}")
    end)
    |> Stream.run()

Now the load takes more than 2 minutes to finish. I don’t understand why, they should be equivalent (I’m loading 10000 entities in both cases).

16

Hmm…I don’t see how that could impact it one way or the other.

17

I think I saw the issue, it is probably on my end, I will test it out and give you feedback

18

Yep, it was my error, basically I’m not using sort, so the list of entities are different in both cases, the one in stream had more than 350000 records hahaha.

1 Like
19

@sezaru I’ve just pushed a pretty significant optimization of relationship and calculation loading (really a complete refactor) to main. You may want to check it out. Make sure to update ash_postgres and ash_graphql to main if you do. It will be released once we’ve had some time to vet it in more projects.