How can I rate limit a queue in Oban?

Questions / Help
1

Here’s my config:

config :my_app, Oban,
  repo: MyApp.Repo,
  # Setting unique period to 604800 in seconds ==  1 week.
  plugins: [{Oban.Plugins.Pruner, max_age: 604_800}],
  queues: [default: 1]

How can I configure a rate limit so that each job is processed at 1 per second at maximum?

2

Get Oban Pro, as the community edition does not have such feature

https://getoban.pro/docs/pro/0.12.9/overview.html

3

I’ll purchase it since it’s a useful feature. Can you link to the specific rate limit in the Oban Pro docs? Would love to learn more.

Edit: Nevermind found it: Smart Engine — Oban Pro v0.12.9

Since we’re just about to launch and not making money, could I just Process.sleep in the worker?

4

That wouldn’t limit concurrency to 1 per second globally though, assuming that’s what you want.

5

Assuming I only had 1 server on my PaaS, it would be global yes?

6

That wouldn’t limit concurrency to 1 per second globally though, assuming that’s what you want.

I think that if you use global_limit, it sets the concurrency across all nodes.

7

@sorentwo pinging you since you asked me to last time I had an Oban question. Appreciate your guidance!

8

To get truly global rate limiting you need to use Pro’s SmartEngine with rate_limit options.

Limiting concurrency to 1 job at a time will work most of the time if you only have one node. If you do rolling deploys or run multiple nodes then you’ll have a global concurrency limit greater than 1.

Global concurrency and rate limiting is complicated!

2 Likes