sezaru
Slow Ash insert/bulk_create with embedded resources
So, I’m importing millions of rows in my system to a resource that has some fields (around 7 fields) that are embedded resources.
I’m inserting the fields using bulk_create with chunks of 10_000 rows.
I noticed that this operation was kinda slow, it takes around 36 seconds to insert the 10_000 rows.
I suspected that the issue could be because of the embedded resources, so I removed them and replaced with fields directly in the main resource.
After that, the same bulk_create call takes 6 seconds to do the same work.
So, it seems that a embedded resource creates a big overhead when inserting a bunch of data.
Zack, do you think there is something that can be done to improve that, or that overhead is to be expected?
Most Liked
barnabasJ
I had a similar issue. I had a calculation in which I made an API request and used a resource with Ash.DataLayer.Simple to convert the data in the response into a list of Ash Resources to have nice Graphql types.
The Resource in question also had a couple of embedded resources and also nested embedded resources. This was also really slow.
The only way I was able to get it to a reasonable timeframe was to do all the conversions myself and create the structs for the embedded resources myself before passing the data to API.bulk_create with the assume_casted?: true option.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









