As far as I understand Partitioned Rate-Limiting, the limit is per partition.

Yes, I realize it’s per partition (we partition on the :profile arg and the API we’re calling is rate limited per profile), but what I’m asking is what it’s actually counting. Is it counting the number of jobs started in a given window, or the number of jobs which ran in a given window (even if they were started outside that window).

It is counting jobs started during that window. Limiting concurrent execution requires the global limit instead.

It sounds like there is a mismatch between the number of allowed jobs, the rate limit, and the time it takes to execute a job.

Related question: if we strictly need f.ex. maximum 100 tasks executed per second (again, 3rd party API rate limit), is this firmly in the Oban Pro territory or can it be achieved with the free engine as well? So far I’ve just opted for specifying 100 jobs for my queue which in my mind means only 100 ever would be picked from the queue (even if it has e.g. 2000 enqueued) and executed concurrently / in parallel – but I might have misread the docs.

Would you please shed some light on this?

This is good to know, though it does bring up the question of how to rate limit the number of jobs that ran in a given window rather than the number of jobs that were started in a given window.

My solution, was to use a global limit of 20 + a rate limit of 70 which should ensure no more than 90 job run in any given window (at most, 20 are started before and 70 started during the window = 90). This has two major downsides.

1. The global limit is, well, global, whereas my rate limit is per partition. This means, if I have, say, 40 partitions, and each are getting 1 job per second, and my jobs take 1 second to run, on average, I’m going to fall behind despite none of them being anywhere near their partition’s rate limit! I would at least like to be able to have a per partition concurrency limit.

2. Even if I could do a per partition concurrency limit, I still have to accept lower max throughput because, in order to ensure I never have more than X jobs run in a given window, I have to set my rate limit to X - my concurrency limit to account for the possibility of jobs that were started, but did not finish, before the window started.

Is there any plans to introduce window based concurrency limiting (I won’t call it rate limiting since that has a different meaning)? How would you suggest achieving the goal of only ever having X jobs run in a given window per partition without reducing max throughput?

My ideal (and how I thought rate limiting worked) was that I could configure a queue to limit concurrency over a window to a given number for a given partition. E.g.

concurrency_rate_limit: [
  # only 90 different jobs will run during any given window, this includes jobs that started, but did not finish, before the window started
  allowed: 90
  period: 10,
  partition: [
    fields: [:args],
    keys: [:profile]
  ]
]

Global limits do apply per-partition, but the issue is with combining the rate limit and global partitions. It’s prevented in config now because while technically possible, the combination melts my brain.

Have you considered bumping the period to match the amount of time jobs take to run? Meaning, instead of 90 per 10 seconds where jobs take 20 seconds to run, make it 45 every 20 seconds to approximate the runtime.

Anything that Pro does can be achieved using the OSS version, but you’ll need to do the heavy lifting yourself and use a separate rate limiter. That means you rate limit in your application somehow and snooze jobs that are over the rate limit. It’s then extra effort to make that global, plus it causes churn while jobs transition between states (executing → scheduled → available) in a loop.

That’s exactly how it works, for a single node. Each node running the queue could run 100 jobs in parallel. If you do a rolling deploy, or run multiple nodes, you’ll exceed that limit.

Thank you. For now I am on a single node so this is good enough. Appreciate you taking the time to make it clear.

It’s prevented in config now because while technically possible, the combination melts my brain.

Fair enough.

Have you considered bumping the period to match the amount of time jobs take to run? Meaning, instead of 90 per 10 seconds where jobs take 20 seconds to run, make it 45 every 20 seconds to approximate the runtime.

The issue is that my jobs are lumpy in the amount of time they take. Basically, we use Finch to make our HTTP calls. Sometimes the TCP connection Finch uses for the HTTP request is closed. When this happens the call fails. We use Tesla retries when this happens to ensure the request goes through.

What this means is that 99% of the time our request will fire basically instantly after the job starts, but sometimes it will have to wait for 1 or even 2 retries to actually fire which can mean it runs multiple seconds after the job starts.

I considered doing away with Tesla retries and just letting Oban retry the job, but we’re using Relay with these jobs, and so if I do that, Relay will broadcast a message saying the job failed even though it really just needs to be tried on a new Finch connection. I realize this is very in the weeds, but basically, these jobs don’t take a consistent amount of time to run, and there’s not a particularly easy way to make them take a consistent amount of time to run.

That’s understandable. The most reliable way to handle this without exceeding the rate limit that I can think of is to increase the window to compensate for the variability, despite the fact that it limits your throughput. Or accept that the limits may not match up perfectly and handle the external rate limit violation gracefully from within the job.

Still thinking about how this could be modeled or even described accurately.