sezaru
I have a requirement for my system where I expect to run jobs that will just call external APIs and then store the data.
I don’t have control on how much time these APIs calls can take, so technically, I can have an API call that will just wait for, I don’t know, 30 minutes before it returns something back.
This means that, if I use Oban to run these as oban jobs, I can have multiple jobs basically “doing nothing” just waiting for a reply from the api server.
The issue I’m having is deciding on a queue size for this.
If I have a small number of concurrent jobs in my queue, then I risk filling up all the available slots with “slow jobs” that take a log time to finish and blocking other jobs from starting.
At the same time, if I make the queue size very large, I think I will have issues with DB connection since, AFAIK, Oban requires a connection for queue slot, at least that’s what I got from this:
Consider the total concurrency across all queues. For example, if you have 4 queues with limits of 10, 20, 30, and 40, your system needs to handle up to 100 concurrent jobs, each potentially requiring database connections and other resources.
It’s not clear to me if the “requiring database connections” here just means that, the code that runs inside the job can require a DB connection, or if it actually means that Oban itself creates that connection when the job is running.
So, can you give me some suggestions on a good strategy for this?
Where Next?
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
sorentwo
If you don’t want those slow or hanging API calls to block the queue, you can use a timeout to ensure the job doesn’t block the queue and retry it later.
The concurrency limit on your queue has little bearing on the total number of DB connections. A connection is only needed for fetching and acking jobs (and Pro does that with a single transaction).
You nailed it—it means that the code running inside the job may require DB connections. Oban does not hold a connection or transaction while the job is running. Doing so would be horrible for concurrency.
Feel free to set a higher concurrency limit (50, 100, whatever suits you). You should also consider a timeout to prevent the queue from clogging after a batch of slower API calls though.