ug0
Problem with testing bulk inserts with Task.async_stream
I have code to do bulk insert which looks like this:
stream = build_attrs_stream()
Repo.transaction(fn ->
stream
|> Stream.chunk_every(5000)
|> Task.async_stream(fn batch ->
{n, _} = Repo.insert_all(Assignment, batch)
n
end, ordered: false)
|> Enum.reduce(0, fn {:ok, n}, acc -> acc + n end)
end)
The above code works fine until I run the test, which gives me error:
(EXIT from #PID<0.544.0>) an exception was raised:
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 939ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
I’m using Phoenix v1.4 and Ecto v3.1, database is Postgres.
Any help?
Most Liked Responses
benwilson512
Hey @ug0 this is because of how the ecto sandbox works during tests. Each test process is provided its own database connection. The Task.async_stream call spawns new processes, and ecto doesn’t know which connection to use. You can solve this by giving Ecto a little more information:
stream = build_attrs_stream()
parent = self()
Repo.transaction(fn ->
stream
|> Stream.chunk_every(5000)
|> Task.async_stream(fn batch ->
{n, _} = Repo.insert_all(Assignment, batch, caller: parent)
n
end, ordered: false)
|> Enum.reduce(0, fn {:ok, n}, acc -> acc + n end)
end)
The change here is the addition of caller: parent as options to the Repo call.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"