DBConnection.ConnectionError client exited on large import

Phoenix Forum Questions / Help
,
1

I am uploading a csv containing 60k records and importing them into postgres db using Ecto. It works fine on local. However, on production, I get this error in the log after importing several thousand records and I get 502 bad gateway.


04:39:56.165 [info] Postgrex.Protocol (#PID<0.3359.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.3516.0> exited

What I tried

  1. SSL connection to db disable/enable
  2. Increased pool size from 5 to 10
  3. Change log level to :debug and I don’t see any other helpful information

Any help is appreciated.

2

Might be worth looking at the following: https://hexdocs.pm/ecto/Ecto.Repo.html specifically the :timeout option for your config?

Sorry if that’s not right, I am still learning a lot myself.

Edit: you might also find this useful: https://hexdocs.pm/ecto/2.2.8/Ecto.Adapters.SQL.Sandbox.html again, specifically ownership_timeout

2 Likes
3

It is probably exceeding the timeout. How are you loading the 60k records? Are you doing 60k individual inserts? Ideally you want to use insert_all for this kind of thing and insert in batches of ~2000.

2 Likes