Hi,

reviving this ancient topic as the error is the same, the solution would probably also work but doesn’t quite feel right. I’d rather try to understand why this is happening.

We’re also experiencing this error in a production environment. It happens rarely (maybe every 2-3 days), often comes in batches of 1-3, and does not cause any other exceptional output in our logs.

DBConnection.ConnectionError: connection not available and request was dropped from queue after 2290ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. By tracking down slow queries and making sure they are running fast enough
  2. Increasing the pool_size (albeit it increases resource consumption)
  3. Allow requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

(db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
(stdlib 3.12.1) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib 3.12.1) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib 3.12.1) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Now to the interesting bits:

• The server does not have any regular traffic (we’re pre-launch), and we’ve seen the error happen at night time as well, so these are quite likely not caused by user requests.
• My main idea for a culprit: For scheduled (cron) jobs, we have Oban running. These jobs are also likely to checkout a DB connection.
• Otherwise the application should be more or less asleep, at least judging from our code. We also have plenty of DB connections available, so in theory the pool should not run out of them.
• I don’t really understand the stacktrace above: :gen_server.try_dispatch/6 is where GenServer sends the message to the callback module, I think. However, DBConnection does not use GenServer and its run/3 is certainly not a GenServer callback. It seems like there are some frames missing in the stacktrace?!

Does anyone have an idea how to debug this better?

[edit] Found this thread with the same error and Oban, but as said we have plenty of DB connections. POOL_SIZE is large.

I am actually using utc_datetime_usec in my app to capture when the user clicked the enrol button, and then sort records based on that - the idea is maybe it’s more fair than looking at database row number. plus it looks super cool when you have a list of people in class sorted by time with microsecond precision

EDIT> sneak peek

Screenshot 2022-01-08 at 1.51.201292×178 45.8 KB