Slow Ash insert/bulk_create with embedded resources

Ash Forum Ash Questions / Help
,
1

So, I’m importing millions of rows in my system to a resource that has some fields (around 7 fields) that are embedded resources.

I’m inserting the fields using bulk_create with chunks of 10_000 rows.

I noticed that this operation was kinda slow, it takes around 36 seconds to insert the 10_000 rows.

I suspected that the issue could be because of the embedded resources, so I removed them and replaced with fields directly in the main resource.

After that, the same bulk_create call takes 6 seconds to do the same work.

So, it seems that a embedded resource creates a big overhead when inserting a bunch of data.

Zack, do you think there is something that can be done to improve that, or that overhead is to be expected?

2

This definitely strikes me as something that should be fixable. It’s possible that the overhead is in the data layer itself, but I think that is probably unlikely. I would need to investigate further, and may not have time to do that this week. You can confirm that the performance issue is Ash by orchestrating a similar operation with bare ecto if you have some time :slight_smile:

3

I will try that when I have some free time and report back to you :smiley:

4

So, someone else reported a similar issue with casting embedded resources. I hope to have some time to look into it later this week.

5

I had a similar issue. I had a calculation in which I made an API request and used a resource with Ash.DataLayer.Simple to convert the data in the response into a list of Ash Resources to have nice Graphql types.

The Resource in question also had a couple of embedded resources and also nested embedded resources. This was also really slow.

The only way I was able to get it to a reasonable timeframe was to do all the conversions myself and create the structs for the embedded resources myself before passing the data to API.bulk_create with the assume_casted?: true option.

1 Like
6

Alright, beginning to look into this issue. There is definitely a significant slow down on embedded resource casting

Name             ips        average  deviation         median         99th %
maps         23.69 K      0.0422 ms    ±56.44%      0.0393 ms       0.107 ms
embeds        0.25 K        4.06 ms     ±4.56%        4.03 ms        4.66 ms
7

Okay so I’ve made some performance improvements. They are still much slower than raw maps, but if you guys could try it out and report back, that would be great. Performance improvements are in main. If you update ash to main, make sure to update ash_postgres and ash_graphql also.

8

After getting everything working with ash/main again, I can say that our calculation is now about 3x faster than before.

9

Thats progress! Are you still having to manually create the embedded resources?

10

I do for now, as I’m still using assume_casted?: true for the bulk_create. Casting ist still taking up too much time in this specific instance.

I just finished upgrading, so I haven’t done any extensive testing yet but here is what I have right now.

I have these 2 reads, the first one is what I tried first but wasn’t fast enough.

I did a little testing with benchee, running the actions with the same list of 7 options each, so I don’t think bulk_create should a lot of optimization there.

Reading with doing the casting:

    read :read_options do
      argument :options, {:array, :map} do
        allow_nil? false
      end

      prepare before_action(fn query ->
                Simple.set_data(
                  query,
                  Ash.Query.get_argument(query, :options)
                  |> Enum.map(&__MODULE__.from_option!/1)
                )
              end)
    end

Takes around 10ms

vs

Preparing the input myself + doing this read action

      prepare before_action(fn query ->
                Simple.set_data(
                  query,
                  Ash.Query.get_argument(query, :availabilities)
                  |> Api.bulk_create(
                    __MODULE__,
                    :create,
                    assume_casted?: true,
                    return_records?: true,
                    return_errors?: true,
                    authorize?: false,
                    max_concurrency: 10
                  )
                  |> then(fn %Ash.BulkResult{records: records, errors: errors} ->
                    records
                  end)
                )
              end)
    end

Takes around 1ms

The thing with assume_casted?: true is that the embedded resources already need to be the real Ash.Resource structs. It might be nice to have an option where it would turn the maps into the structs but wouldn’t cast the values for it.

11

Interesting…I definitely want to track down the main slow down. I think there is likely just some unnecessary work being done there.

12

Sidenote: The structure uses a lot of embedded resources.

it does look something like this.

%{
  attribute1: value,
  embedded_resource1: %{
    attribute1: value,
    attribute2: value,
    embedded_resource1: %{},
    embedded_resource1: %{},
    embedded_resource1: %{
      embedded_resource1: %{},
      embedded_resource2: %{},
      embedded_resource3: %{},
      embedded_resource4: %{}
    },
    embedded_resource2: %{
      embedded_resource1: %{},
      embedded_resource2: %{},
      embedded_resource2: %{},
      embedded_resource2: %{}
    }
  },
  embedded_resource2: %{
    attribute1: value,
    attribute2: value,
    embedded_resource1: %{
      attribute1: value,
      attribute2: value,
      attribute3: value,
      embedded_resource1: %{},
      embedded_resource2: %{}
    }
  }
}

So there is a lot of casting going on.