Timeout errors with long running transactions

OK, after downgrading everything just works. Thanks for the hint, @outlog and @engineeringdept!

Huh…

{:bad_action_from_state_function, {:reply, :undefined, {:error, :timeout}}}

I actually get this error nightly since a few days. I had not yet the time to look into this. So it seems very likely that you are right.

Phew, you just solved me some headaches…

think you are hitting this 21.3 OTP bug that is just emerging:
and mentioned here Workaround for Docker with Erlang/OTP 21.3 and

Redirecting… (identical but with patch)

can you downgrade to OTP 21.2.7 ? or wait for patch release (presumably later today)..

I’m not using distillery (had not yet the time to wrap my head around that), I just checked out the source and compiled it on the server. Using mix phx.server via systemd to start the service.

ckruse@ares:/var/www/termitool$ mix hex.outdated
Dependency        Current  Latest  Update possible
absinthe_phoenix  1.4.3    1.4.3
absinthe_plug     1.4.6    1.4.6
appsignal         1.9.4    1.9.4
arc               0.11.0   0.11.0
arc_ecto          0.11.1   0.11.1
bamboo            1.2.0    1.2.0
bamboo_smtp       1.6.0    1.6.0
bcrypt_elixir     1.1.1    2.0.1   No
cachex            3.1.3    3.1.3
comeonin          4.1.2    5.1.1   No
csv               2.2.0    2.2.0
dataloader        1.0.6    1.0.6
ecto_sql          3.0.5    3.0.5
ex_phone_number   0.2.0    0.2.0
exldap            0.6.3    0.6.3
gettext           0.16.1   0.16.1
guardian          1.2.1    1.2.1
jason             1.1.2    1.1.2
phoenix           1.4.2    1.4.2
phoenix_ecto      4.0.0    4.0.0
phoenix_pubsub    1.1.2    1.1.2
plug              1.7.2    1.7.2
plug_cowboy       2.0.1    2.0.1
porcelain         2.0.3    2.0.3
postgrex          0.14.1   0.14.1
timex             3.5.0    3.5.0

Nope. Bare metal with Ubuntu 18.04 LTS

ckruse@ares:/var/www/termitool$ erl -version
Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 10.3
ckruse@ares:/var/www/termitool$ elixir --version
Erlang/OTP 21 [erts-10.3] [source] [64-bit] [smp:48:48] [ds:48:48:10] [async-threads:1] [hipe]

Elixir 1.8.1 (compiled with Erlang/OTP 20)
ckruse@ares:/var/www/termitool$

fishing expedition here..

but check OTP and elixir versions and make sure that you are on latest patch release of what you are using.. (recently there was a transient bug with ConnectionError fixed in later OTP..)

also can you check/post mix hex.outdated for the relevant dependencies..

also is this using distillery and what version?

Are you running in a Docker container? I discovered a bug in the the official Elixir 1.8.1 container, which is backed by Erlang 21.3, which caused SSL problems like these in production. I had to switch to a custom image running Erlang 21.2.7.

Other applications using ecto on the same server work like a charm… none of my queries seem to take more than 20ms, most of them below 10ms. The :pool_size is 50. I’m seriously confused.

Dammit.

The errors appeared again, random connection drops by Postgrex:

11:38:50.415 [error] Postgrex.Protocol (#PID<0.425.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
11:38:52.415 [error] Postgrex.Protocol (#PID<0.431.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
11:38:54.415 [error] Postgrex.Protocol (#PID<0.418.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
11:38:56.415 [error] Postgrex.Protocol (#PID<0.424.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
11:38:58.415 [error] Postgrex.Protocol (#PID<0.433.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
11:39:00.415 [error] Postgrex.Protocol (#PID<0.426.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed

And this time there was not even an import job running, I just removed the timeout: :infinity from the repo configuration.

What’s going on?!

Hm. Race conditions on the data are a non-issue in this case. But I will overhaul my import pipeline, thanks for your input…

That said, I guess I found the underlying cause for this problem. Your hint with concurrently importing the data gave me the idea: the CSV library uses workers to parallelize the CSV reading and parsing. That means that my Stream.map() executions are parallelized as well.

Parallelized execution means: different ecto processes are used. And since the server has traffic and more cores than my workstation this means: more Stream workers, more user connections and thus the pool (I was using the default size of 15) could be exhausted pretty fast.

And indeed, if I reduce the pool size on my workstation to 2 and the timeout to 1 second, I get the same errors all over the place.

To avoid that I now use the :caller option on all repo calls, and now it works like a charm on my dev machine. Have still to test it in production, though

This also means: my import did not run in the transaction anyways…