rathoud96

rathoud96

All Oban queue producers crash simultaneously due to ObanRepo pool exhaustion on Cloud Run (VPC idle timeout)

Environment

  • Elixir 1.18.3-otp-27 / OTP 27.3.4
  • Oban 2.20.2
  • Phoenix 1.7.x
  • db_connection 2.8.1 / Postgrex 0.21.1
  • Infrastructure: Google Cloud Run (serverless, min 1 instance, max 10) behind a private VPC (vpc-egress=private-ranges-only)

Setup

We run Oban on a dedicated ObanRepo (separate from our main Repo) with the following config:

runtime.exs

config :myapp, ObanRepo,
url: database_url,
pool_size: 5,
prepare: :unnamed,
idle_interval: 15_000,
connect_timeout: 10_000,
socket_options: \[keepalive: true\]

config :myapp, Oban,
repo: ObanRepo,
peer: Oban.Peers.Postgres,
notifier: Oban.Notifiers.PG,
queues: \[
metadata_discovery_high: 3,
metadata_download_high: 1,
metadata_discovery: 2,
metadata_download: 1,
metadata_enrichment: 1,
metadata_search: 1,
token_refresh: 2,
default: 5
\]

Problem

Roughly once a day (seemingly unprovoked — no active retrieval jobs running), all Oban queue producers crash simultaneously and Oban becomes non-functional until the instance restarts.

The failure always follows the same cascade:

Step 1 --- SSL connections drop silently:
\[error\] Postgrex.Protocol (#PID<0.3205.0>) disconnected:
\*\* (DBConnection.ConnectionError) ssl recv (idle): closed

\[error\] Postgrex.Protocol (#PID<0.3206.0>) failed to connect:
\*\* (DBConnection.ConnectionError) ssl send: closed

Step 2 --- Postgrex reconnection attempts time out:
\[error\] Postgrex.Protocol (#PID<0.3209.0>) timed out because it was
handshaking for longer than 15000ms

Step 3 --- Every queue producer terminates:
\[error\] GenServer {Oban.Registry, {Oban, {:producer, "metadata_discovery_high"}}} terminating
\*\* (DBConnection.ConnectionError) connection not available and request was
dropped from queue after 700ms.

\[error\] GenServer {Oban.Registry, {Oban, {:producer, "metadata_download"}}} terminating
\*\* (DBConnection.ConnectionError) connection not available and request was
dropped from queue after 5201ms.

…same for all 8 queues

Step 4 --- Peer loses leader election:
\[warning\] Oban.Peer.leader?/2 check failed due to
{:timeout, {GenServer, :call, \[#PID<0.3276.0>, :leader?, 5000\]}}

Questions

  1. Is poll_interval the right lever here? With Oban.Notifiers.PG handling real-time wakeups, is there any meaningful downside to a 30-second poll interval beyond a max 30-second delay on missed notifications?
  2. What is the recommended minimum ObanRepo pool size for a setup with Oban.Peers.Postgres + Oban.Notifiers.PG + 8 queues? We’re trying to right-size rather than just throw connections at it.
  3. Is there an Oban-level setting for environments with network-enforced idle timeouts (serverless/VPC) that we’re missing — beyond idle_interval (which only fires every 15s, potentially too slow) and
    socket-level keepalive: true?

Any guidance appreciated — especially from folks running Oban on GCP Cloud Run or similar ephemeral/serverless infrastructure.

Most Liked

sorentwo

sorentwo

Oban Core Team

There are numerous improvements to safe querying in Oban v2.21 and v2.22 that should help with frequent queries causing a failure cascade.

There isn’t a poll_interval option anymore, it was renamed to stage_interval long ago. You could set that to a reduced rate and have fewer intermittent queries. However, it would change the granularity of scheduled jobs (effectively at most every 30 seconds rather than down to the second).

That depends entirely on throughput for those queues. You could easily run that with 10 connections. The issue you’re encountering is about a missing database, not pool exhaustion.

Nothing in particular that I’m aware of. Oban is designed for long-running processes with a consistent database connection. Using it in an ephemeral environment is bound to cause some issues.

Where Next?

Popular in Troubleshooting Top

rayex
When I compile hex package io_ansi_plus, I get error “Codepoint failed”. Why? and why are the “Got:” and “Hint:” codepoints identical? It...
New
anotherpit
Summary A downstream job that deps on a graft correctly waits for the grafted jobs at one graft level, but stops waiting when the workflo...
New
rathoud96
Environment Elixir 1.18.3-otp-27 / OTP 27.3.4 Oban 2.20.2 Phoenix 1.7.x db_connection 2.8.1 / Postgrex 0.21.1 Infrastructure: Google Cl...
New
hyperoceanic
Having read the book, I’m keen to explore Ash a bit more, but I’m falling at the first hurdle - the creation of a new project. Here’s my...
New
ktayah
Environment Oban Pro: 1.7.6 Oban: 2.22.1 Issue When a sub-workflow built with Workflow.put_context/2 is attached to a parent via ...
New
tellemiller
We recently started running Oban with Oban Web on PostgreSQL (AWS RDS gp3) and noticed our oban_jobs buffer cache hit ratio sitting at 65...
New
AstraLuma
Experienced programmer writer her first Elixir, and maybe bit off more than she can chew? I’m trying to start two GenServers, where the ...
New
anotherpit
A graft inside another graft causes Oban.Pro.Workflow.status to recurse forever. oban_pro 1.7.5 defmodule App.NestedGraftRepro.Test do ...
New
Lotoen
Hi, I’m on Linux Mint 22.3 - Cinnamon 64-bit, and I encountered an error when trying to build an elixir phoenix container 577.1 Reading...
New
hakarabakara1
I have three modules, a Business which is associated Tags and Categories though join tables business_tags and business_categories. I inte...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New

We're in Beta

About us Mission Statement