vlymar
I’d appreciate some help proving (or disproving) a hypothesis I have about Oban’s overhead when run on multiple nodes.
Background
-
oban 2.17.12
-
oban_pro 1.14.10
-
oban_notifiers_phoenix 0.1.0
-
engine: Oban.Pro.Engines.Smart
-
notifier: Oban.Notifiers.Phoenix
-
database: AWS Aurora RDS postgresql
Caveat: we’re aware that using Aurora with Oban is not ideal. Moving Oban off of Aurora isn’t a short term option, so we’re investigating opportunities to improve performance on Aurora.
Aurora provides this dashboard that shows a representation of database load by SQL statement. The yellow category of load is IO:XactSync. My understanding is that txn commits force aurora to flush to object storage, so commits are particularly expensive in Aurora compared to vanilla PG (this may be one of the issues with using oban with aurora).
Conclusions from that screenshot:
- our top single source of database load is from a
COMMITissued by Oban- we’ve auto-injected comments into some statements to see where they originate, hence the
-- Switchboard.Repo.ObanWrapper.transaction/2 (L14)comment
- we’ve auto-injected comments into some statements to see where they originate, hence the
- That commit has a very high Calls/sec, dwarfing any of our other top statements in QPS.
- There’s another source of load that has a similar QPS:
WITH "subset" AS (SELECT ...I found this query in the smart engine - it’s called byfetch_jobsand is used to fetch a batch of jobs and mark them as executing.
Here’s my hypothesis: For every job insertion (or insertion batch) Oban notifies every k8s pod running Oban. Upon receiving the notification, each pod running Oban calls fetch_jobs (possibly once per queue, though I’m not sure). Each fetch_jobs call generates a transaction commit. Committing transactions is quite expensive with Aurora, so what should be a relatively lightweight (albeit high frequency) operation is disproportionally expensive in Aurora-land. Furthermore, we’ve been using k8s pods as a convenient unit of scaling throughput for job processing. But if this hypothesis is correct, adding pods significantly increases load on our DB, creating a bottleneck.
To test this hypothesis, I played with our staging environment’s pod count and saw notable fluctuations in commit volume that matched changes in pod count (even though the rate of job processing did not change).
My conclusion is that if we want to keep the low latency triggering afforded by Oban Notifiers, we should try running fewer, beefier pods. We could tune the system to have the same total number of workers & cpu cores working on fewer pods. Fewer pods would lead to a lower rate of fetch_job calls, and fewer commits to Aurora.
Alternatively, we could investigate some sort of debounce in the Notifier, or switch off the notifier to polling.
Would love to read thoughts on this. Am I misunderstanding any things about how Oban or Aurora work? Does it make sense that running fewer oban nodes would reduce our commit volume (by running fewer of those WITH "subset"...FOR UPDATE SKIP LOCKED transactions). Are we missing any key performance improvements from later versions of Oban?
Besides this, does anyone have thoughts to share on optimizing Oban throughput on Aurora?
Thanks for the help!
Where Next?
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #ai
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Most Liked- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
sorentwo
That’s fascinating. It explains a great deal about why Aurora instances have such IO issues with Oban compared to other database types. Previously, I thought it was due to a lack of unlogged tables and IO throttling.
That’s true if the
insert_triggeroption is enabled, which is the default. You can disable the trigger at the expense of less responsive insert handling (up to a second instead of sub-second).There’s also the
dispatch_cooldownoption to control how long a producer waits between subsequent fetches. The default is 5ms, and the goal is to prevent thrashing the database with rapidfetch_jobsrequests. You can tune the value to force fewer fetch requests at the expense of lower throughput.That seems like a great idea, and worthy of being included in the Troubleshooting guide for Aurora users.
That makes sense to me. Fewer nodes, combined with tweaks to insert notifications and dispatch cooldown, could reduce the total number of
COMMIToperations tremendously. Something important to note is thatSKIP LOCKEDrequires a transaction or it has no effect.No you aren’t. I’m not sure there’s anything else Oban can do here (well, not much more Pro can do, it has much more optimized acking/fetching compared to OSS).
Please report back if/when you make changes and let us know how it goes!
brentjanderson
Adjacent to this, Postgrex recommends setting the
:endpointsparameter when using Aurora. This may be useful in the Oban Troubleshooting guide as well, as we have found that Oban requires manual intervention to recover from an Aurora failover. When using this Postgrex:endpointsfeature, however, Oban recovers automatically.vlymar
Quick update:
enabling Aurora IO-optimized mode reduced our commit latency which in turn reduced the impact of Oban’s high frequency of commits in our system.
we’re saving
dispatch_cooldownand pod count tuning for the new year, will report back when we have some results