Is there any limitations in Oban queue size with the DB?

[sezaru]

I have a requirement for my system where I expect to run jobs that will just call external APIs and then store the data.

I don’t have control on how much time these APIs calls can take, so technically, I can have an API call that will just wait for, I don’t know, 30 minutes before it returns something back.

This means that, if I use Oban to run these as oban jobs, I can have multiple jobs basically “doing nothing” just waiting for a reply from the api server.

The issue I’m having is deciding on a queue size for this.

If I have a small number of concurrent jobs in my queue, then I risk filling up all the available slots with “slow jobs” that take a log time to finish and blocking other jobs from starting.

At the same time, if I make the queue size very large, I think I will have issues with DB connection since, AFAIK, Oban requires a connection for queue slot, at least that’s what I got from this:

Queue Planning Guidelines - Defining Queues — Oban v2.20.3

Consider the total concurrency across all queues. For example, if you have 4 queues with limits of 10, 20, 30, and 40, your system needs to handle up to 100 concurrent jobs, each potentially requiring database connections and other resources.

It’s not clear to me if the “requiring database connections” here just means that, the code that runs inside the job can require a DB connection, or if it actually means that Oban itself creates that connection when the job is running.

So, can you give me some suggestions on a good strategy for this?

[sorentwo]

@sezaru

I don’t have control on how much time these APIs calls can take, so technically, I can have an API call that will just wait for, I don’t know, 30 minutes before it returns something back.

If you don’t want those slow or hanging API calls to block the queue, you can use a timeout to ensure the job doesn’t block the queue and retry it later.

@sezaru

At the same time, if I make the queue size very large, I think I will have issues with DB connection since, AFAIK, Oban requires a connection for queue slot, at least that’s what I got from this:

The concurrency limit on your queue has little bearing on the total number of DB connections. A connection is only needed for fetching and acking jobs (and Pro does that with a single transaction).

@sezaru

It’s not clear to me if the “requiring database connections” here just means that, the code that runs inside the job can require a DB connection, or if it actually means that Oban itself creates that connection when the job is running.

You nailed it—it means that the code running inside the job may require DB connections. Oban does not hold a connection or transaction while the job is running. Doing so would be horrible for concurrency.

@sezaru

So, can you give me some suggestions on a good strategy for this?

Feel free to set a higher concurrency limit (50, 100, whatever suits you). You should also consider a timeout to prevent the queue from clogging after a batch of slower API calls though.