Distillery app fails to start randomly, database connection issue?

New deployment today got me confused. Three instances of the same Elixir app on the same server, Postgres DB on a second machine. App is packaged with Distillery, each instance has its own slightly different config.

First one starts fine with bin/app start. Second and third one … don’t. Hmm. I run bin/app foreground and bin/app console instead – and both run fine! bin/app start again … nothing.

An erl_crash.dump appears. It says:

Thu Nov 19 10:37:41 2020
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,db_connection,{{shutdown,{failed_to_start_child,'Elixir.DBConnection.Ownership.PoolSupervisor',{'EXIT',{badarg,[{io,put_chars
System version: Erlang/OTP 23 [erts-11.0.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Compiled: Mon Jul 27 12:43:05 2020
Taints: crypto,asn1rt_nif,eiconv,dyntrace

So some database connection problem? Unfortunately the error message gets cut off, and since bin/app console starts fine, there’s no way for me to see it in its entirety.

Out of frustration, I run bin/app start like five or ten times, and suddenly … pong! It randomly starts working. Third instance of the app – exactly the same story.

Now I’m glad it works, and it’s been running smoothly all day, but what is going on here? What could the database error mean? Is this some connection pool issue that could break on me any minute? Where do I learn more? Thank you all for your ideas :slight_smile:

1 Like