nathanl

nathanl

We were seeing bugs where 1) jobs would sit indefinitely in “available” and 2) the Oban Web dashboard would sometimes display a count of 0 for all queues.

It seems the root cause for both was:

  • Oban.Notifiers.Postgres uses Postgrex.SimpleConnection
  • If the connection to PostgreSQL “goes bad” (stays connected, but stops
    responding)…
  • Then Postgrex.SimpleConnection would seem to reconnect but
    actually would not establish a new connection to PostgreSQL (as shown
    in the pg_stat_activity table)…
  • and from then on, that node would not run Oban jobs and its Oban Web
    UI would show count 0 for every queue

This issue has been fixed in postgrex as of v0.20.0, released today.

Showing Posts 1 to 1

nathanl

nathanl OP

Details on how we reproduced the bug and confirmed the fix:

  • Modify Oban.Notifiers.Postgres to look for an ENV var, and, if present, connect to PostgreSQL via a different port, proxied by GitHub - Shopify/toxiproxy: ⏰ A TCP proxy to simulate network and system conditions for chaos and resiliency testing · GitHub
  • Start the Phoenix app locally in a 3-instance cluster, each listening on a different HTTP port, and give 2 instances the “use toxiproxy” ENV var
  • In psql, see that there are 3 connections to the database whose last executed query was a LISTEN using SELECT state_change, client_addr, client_port, pid FROM pg_stat_activity WHERE state = 'idle' and datname = 'gridpoint_dev' AND query LIKE '%LISTEN%' ORDER BY state_change DESC;
  • Insert many jobs
  • As they begin to be processed, turn on a “toxic” in toxiproxy, telling it to leave those connections open but block all data from flowing to the client: curl -X POST -H "Content-Type: application/json" -d '{ "name": "immediate_timeout", "type": "timeout", "toxicity": 1.0, "stream": "downstream", "attributes": { "timeout": 0 } }' http://localhost:8474/proxies/postgres/toxics
  • After a few seconds, see that the pg_stat_activity query now shows only 1 connection
  • See that Oban Web on the host not using toxiproxy shows jobs waiting in “available”, but on one of the hosts using toxiproxy, shows 0 counts for all queues. (Note: in a deployed environment with hosts behind a load balancer this is not apparent.)
  • Turn off the “toxic” in toxiproxy: curl -X DELETE http://localhost:8474/proxies/postgres/toxics/immediate_timeout
  • Observe different behavior:
    • With postgrex v0.19.3 (2024-11-12), pg_stat_activity does not show the other 2 connections get reestablished, the “available” jobs remain available, and Oban Web on the formerly toxiproxied host keeps showing 0 for queue counts until/unless the Oban.Notifiers.Postgres process is killed and restarted, at which point that host’s jobs are processed and Oban Web on that host shows correct queue counts
    • With postgrex v0.20.0 (2025-02-05), pg_stat_activity shows the other 2 connections get reestablished, the “available” jobs get processed, and Oban Web on the formerly toxiproxied hosts show correct queue counts

To set up toxiproxy for the usage above:

# start toxiproxy
# Make the control API available on port 8474 and the proxy port on port 8666:
docker run -d --rm --name toxiproxy -p 8474:8474 -p 8666:8666 shopify/toxiproxy

# verify it's running
docker ps | grep toxiproxy

# create a proxy for postgres
curl -X POST -H "Content-Type: application/json" \
  -d '{
    "name": "postgres",
    "listen": "0.0.0.0:8666",
    "upstream": "host.docker.internal:5432"
  }' \
  http://localhost:8474/proxies

# verify you can connect to postgres through it
psql -h localhost -p 8666 -U postgres
— All posts loaded —

Where Next? Top

Trending in Questions Top

katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
achenet
Hello, I’m trying to build a basic Phoenix web-app, and I’d like to use Tailwind. However, when I launch mix phx.server, I get an error...
New
bradley
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
Cxx-mlr
I’m working on a small exercise involving update_in/3, and I came up with this solution: data = %{ name: "Periodic Table", category:...
New
ChrisAmelia
I’ve got trouble wrapping my head around the order in which functions are called in this snippet (from Phoenix’s authentication): toke...
New
unaware8150
Hello folks! So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
dillonoconnor
Is there any way to avoid the Hologram compiler running when using iex? It seems like the front-end code could potentially be disregarded...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
georgeguimaraes
Just published claude-code-elixir, a plugin marketplace for Claude Code with Elixir support. These are the plugins I’ve been using for my...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews