dgigafox

dgigafox

Plan to create a seeder library

Hi guys. Would like to ask for some help cause I am new to creating libraries.
So I am currently creating an elixir library for seeding that would solve this problem:

Seeding a database is simple using seeds.exs but fairly messy and complicated when the app is huge and has a bunch of tables. For example, the project that I am working on currently is taking like 30 to 40 minutes to seed which does not count DB timeouts which you need to re-run again if it happens.

Now the interface of my library is simple. Seeders have 2 callbacks:

@callback deps() :: [atom()]
@callback run(Context.t()) :: struct() | [struct()]

deps are list of seeders that should be run before the current seeder while run returns a struct/structs. It will look something like this

defmodule UserSeed do
  use ThisLibrary, key: :user

  @impl true
  def deps, do: []

  @impl true
  def run(%ThisLibrary.Context{}) do
    {:ok, users} = ...some_insert_operation()
    users
  end
end

After this seed runs it will be put to Context.seeds with the key which can be pattern matched by other seeders next in line. Something like:

defmodule PostSeed do
  use ThisLibrary, key: :post

  @impl true
  def deps, do: [UserSeed]

  @impl true
  def run(%ThisLibrary.Context{seeds: %{users: users}}) do
    {:ok, posts} = ...some_insert_operation(users)
    posts
  end
end

Now, I would like my implementation to be using GenServers. Each seeder library is a GenServer periodically checking if all deps already ran. If all deps run then that’s the time it will call run/0 and do the DB operation. The library runs a dynamic supervisor that starts each seeder as a child. So in case of error it can save the state in whatever format (maybe database, json file, etc…), re-running the seeder library will not start from the beginning but will just run from where it stopped. Also would like to run seeders asynchronously as long as its deps are complete for faster seeding.

Question is. Is this a good implementation? Am I overcomplicating things, or am I missing something?

Most Liked

al2o3cr

al2o3cr

Roughly how many rows are involved with this seed? 30-40 minutes seems like it should be tens of millions, which seems excessive.

Hitting DB timeouts suggests two things could be happening:

  • inserting too many rows in a single transaction
  • if there aren’t many rows involved, inserting things in ways that cause lock contention (unique indexes, etc)

It’s also worth investigating the database you’re seeding into - is it hitting a performance bottleneck? If so, dividing work up among GenServers is just going to mean more processes waiting for an overloaded DB.

al2o3cr

al2o3cr

30-40 minutes to insert 40k rows seems remarkably slow; I’d recommend you start by figuring out where all that time is going. For instance, changing up your seeding workflow won’t help much if (for instance) you’re using bcrypt_elixir and inserting a lot of users with log_rounds set too high.

If you need updatable seeds, a package like phil_columns could be what you’re looking for.

Where Next?

Popular in Questions Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

We're in Beta

About us Mission Statement