preciz
Oban queues disappearing randomly in production
This is going on for a while now. I think it’s best described with an example:
We have an Oban queue “postgresql” with 100K jobs available.
We deploy production, the nodes start and the queues start processing the jobs.
Then the queues start to disappear (no errors in logs) and the processing is slow-ish, it is still going but on the nodes the queues appear / disappear kind of randomly and at one point usually 1/3 of them are running.
And this happens only with a few queues, usually the ones that have a larger number of jobs waiting.
Anybody ever had this problem?
First Post!
sorentwo
Which Oban and Pro versions? Also, which hosting provider? The only ways I’m aware of that the producers would slow down and potentially crash is from query timeouts, but that would log errors.
Are these inserted all at once, or was it accumulated over time? It shouldn’t change whether the queue stays active, but Pro v1.7 shipped with a feature called automatic spacing, which prevents a big dump of jobs from overwhelming the queue.
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"