AHBruns

How to rate limit with Oban?

We are using Oban rate limiting to ensure we don’t overload an external service. The service has a rate limit of no more than 100 requests in any given 10 second window. We make all requests in Oban jobs, and rate limit the queue like so:

rate_limit: [
        # reserve 10% of our allotted requests for other uses (e.g. manual API calls)
        allowed: 90,
        period: 10,
        partition: [
          fields: [:args],
          keys: [:profile]
        ]
      ]

Somehow we’re still hitting the API’s rate limit. We believe the issue lies in Oban’s rate limiting semantics. Basically, we’re wondering if Oban limits how many jobs can be running in a given window, or if it limits how many jobs can be started in a given window.

E.g. if a queue has a rate limit of 10 jobs with a window of 10 seconds, and 10 jobs are running but they were started an hour ago then when a new job is enqueued will it be started right away or will it wait for one of the currently running jobs to stop?

13 comments

/oban #oban-pro

16 1444 13

2024-06-17 18:52:57 UTC

Marked As Solved

sorentwo

Oban Core Team

It is counting jobs started during that window. Limiting concurrent execution requires the global limit instead.

It sounds like there is a mismatch between the number of allowed jobs, the rate limit, and the time it takes to execute a job.

Post #4

Also Liked

AHBruns

I mean user facing. I think ultimately, all limits should come down to asking

What am I counting? E.g. the number of running jobs on a given queue, the number of jobs that have been started on a given queue, the number of running jobs on a given node, something else?
Over what window am I counting? E.g. a zero length window, a 10 second window, a 2 week window?
What is the max count I’m allowed to hit? Aka the limit.

Every one of the existing APIs can be framed in this context.

a local limit is counting the number of running jobs on on a given node, it’s counting over a zero length window (aka an instant), and the max count is supplied by the user
a global limit is counting the number of running jobs on on a given queue, it’s counting over a zero length window, and the max count is user supplied
a rate limit is counting the number of jobs started in any given supplied partition (or the queue if none is supplied), it’s counting over a user supplied window length, and the max count is user supplied

I have a use case to count the number of running jobs on a given partition over a non zero length window, but I could imagine lots of other useful combinations. Off the top of my head:

counting the number of running jobs on a given partition over a zero length window (aka a partition limit rather than a global or local limit).
counting the number of started jobs on a given node over a non-zero length window (aka a per node rate limit)

The issue comes up in defining the combinations. E.g. what does measuring the number of jobs started over a zero length window mean? If you model time in discrete ticks, then that’s pretty easy, it’s how many jobs were started in a given tick, but if you model time as continuous, then it’s kind of a meaningless question. Imo, saying that your API is only accurate down to some fundamental tick size is pretty reasonable, and makes everything coherent. In such a world a “zero length window” really means a window that is the length of 1 tick.

Post #14

AHBruns

This is good to know, though it does bring up the question of how to rate limit the number of jobs that ran in a given window rather than the number of jobs that were started in a given window.

My solution, was to use a global limit of 20 + a rate limit of 70 which should ensure no more than 90 job run in any given window (at most, 20 are started before and 70 started during the window = 90). This has two major downsides.

The global limit is, well, global, whereas my rate limit is per partition. This means, if I have, say, 40 partitions, and each are getting 1 job per second, and my jobs take 1 second to run, on average, I’m going to fall behind despite none of them being anywhere near their partition’s rate limit! I would at least like to be able to have a per partition concurrency limit.
Even if I could do a per partition concurrency limit, I still have to accept lower max throughput because, in order to ensure I never have more than X jobs run in a given window, I have to set my rate limit to X - my concurrency limit to account for the possibility of jobs that were started, but did not finish, before the window started.

Is there any plans to introduce window based concurrency limiting (I won’t call it rate limiting since that has a different meaning)? How would you suggest achieving the goal of only ever having X jobs run in a given window per partition without reducing max throughput?

My ideal (and how I thought rate limiting worked) was that I could configure a queue to limit concurrency over a window to a given number for a given partition. E.g.

concurrency_rate_limit: [
  # only 90 different jobs will run during any given window, this includes jobs that started, but did not finish, before the window started
  allowed: 90
  period: 10,
  partition: [
    fields: [:args],
    keys: [:profile]
  ]
]

Post #6

sorentwo

Oban Core Team

Anything that Pro does can be achieved using the OSS version, but you’ll need to do the heavy lifting yourself and use a separate rate limiter. That means you rate limit in your application somehow and snooze jobs that are over the rate limit. It’s then extra effort to make that global, plus it causes churn while jobs transition between states (executing → scheduled → available) in a loop.

That’s exactly how it works, for a single node. Each node running the queue could run 100 jobs in parallel. If you do a rolling deploy, or run multiple nodes, you’ll exceed that limit.

Post #8