Problem with parallel run migration with ecto in runtime via async tasks

Questions / Help
, , , ,
1

hi elixir community

I am working to create runtime databases for each user parallel via async tasks, for example, user1, user2, etc
1..count |> Task.async_stream(fn counter -> GenServer.call(String.to_atom("user#{counter}_db_worker"), {:create_dbs, counter}) end, [max_concurrency: 3, timeout: 30_000]) |> Enum.each(fn x -> IO.puts "Batch #{x} done" end)

but When I run the code with max_concurrency: 1 databases and migration are created so it must be a problem with the parallel processing.
any help thanks in advance.
this leads to the following error:

** (CompileError) _build/dev/lib/manager_with_sql/priv/repo/migrations/20220129033040_create_user.exs:1: cannot define module ManagerWithSql.Repo.Migrations.CreateUser because it is currently being defined in _build/dev/lib/manager_with_sql/priv/repo/migrations/20220129033040_create_user.exs:1

2

Migrations are scripts and are not compiled into your app. When you’re trying to run them concurrently, the compiler sees duplicate module definitions. I don’t think Ecto migrations were meant to be run concurrently and I’m assuming you’re going to hit even more issues along the way.

Can you run them serially? How do you pass different configs to the migrations?

3

This might give you some ideas: Dynamic repos - Redefining module warnings · Issue #152 · elixir-ecto/ecto_sql · GitHub

basically, create your migrations as “normal” modules inside of lib/ instead of scripts outside of it. and then they will only load once and you can pass a list of module names/versions into Ecto.Migrator.run