mbklein

mbklein

Have Ecto wait for DB to spin up?

We’re considering using AWS Aurora Serverless PostgresSQL for our developer environment. I’ve already determined that it will meet our needs as far as Postgres compatibility, but I’m looking for help getting over the last inconvenience. Whenever the DB is idle for a certain amount of time, it spins down completely. It spins itself back up when the next connection attempt is made, but it takes ~20-30 seconds. What this means is that when I start my Elixir/Ecto/Postgrex application, I get a whole lot of DBConnection errors. I can then wait 20 seconds and restart and everything works fine.

One other thing that’s probably relevant is that the port is open and the cluster is listening even when it’s spun all the way down. That is, I can do the following:

  • See cluster capacity at 0
  • Run nc -v DB_HOST 5432 on my client, and immediately get Ncat: Connected to IP_ADDR:5432 in return
  • See cluster capacity spinning up to 2 just from that port connection

So I’m guessing that Postgrex isn’t failing to connect; it’s just failing to get a response in a timeframe that makes DBConnection happy.

I’ve set timeout, connect_timeout, and handshake_timeout to 60 seconds, but I’m still getting errors from GenServers and other processes that start up and immediately try to query the Repo, e.g.,

** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2918ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

I have no idea what appropriate queue_target and queue_interval settings would be for this particular case. I’d also love to be able to display some kind of “Waiting for database to spin up…” message.

Any thoughts on this would be greatly appreciated.

Most Liked

joaoevangelista

joaoevangelista

Reading the docs, I got the idea that the queue_ options are for preventing overload, but found a interesting one: after_connect_timeout

The maximum time allowed to perform function specified by :after_connect option (default: 15_000 ).

There is also a configure option, that you can use to print messages and reconfigure before any attempt to connect:

A function to run before every connect attempt to dynamically configure the options, either a 1-arity fun, {module, function, args} with options prepended to args or nil where only returned options are passed to connect callback (default: nil )

Here is the full docs on the options for DBConnect: DBConnection — db_connection v2.10.1

mbklein

mbklein

Ah, so if after_connect blocks, I could use it to wait for the connection to actually return something, with an after_connect_timeout of 60 seconds or so. Worth a try. Thank you!

Where Next?

Popular in Questions Top

chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

We're in Beta

About us Mission Statement