How to rate limit Oban workers by their Workflow?

Special Sections Oban
,
1

Hey folks, I wondered how I could rate limit Oban workers by their workflow so that one Workflow has to complete before the next Workflow starts. Here’s my setup and what I’m trying to achieve:

I have a Workflow with 3 Steps and they all depend on each other: A -> B -> C

So, first A needs to complete, then B and only then C runs. It is important that A or B don’t run again before C completes. I want to run these steps in parallel for multiple organizations, but for each organization the workflows should run sequentially. Every workflow receives an organization_id as an argument. That’s how I can partition the workers.

The sequence would look something like this:

Each [...] is one full Workflow execution

Org1: [A -> B -> C] [A -> B -> C][A -> B -> C]
Org2: [A->B->C][A->B->C][A->B->C]

So, I run the workflow in parallel for each organization, but in sequence per organization.

Would something like this work?

config :my_app, Oban,
  queues: [
    workflows: [
      rate_limit: [allowed: 1, partition: [:workflow, args: [:organization_id]]]
    ]
  ]
2

This may be a good place to use a Chain instead of a workflow. As long as the three A/B/C jobs are inserted together, the chain will guarantee that they run before another group of A/B/C jobs for the same organization:

use Oban.Pro.Worker, chain: [by: [:worker, args: :organization_id]]

Chains for each organization can still run in parallel that way, without any extra concurrency limitations.

Probably not, as rate limiting changes how frequently jobs are fetched, but wouldn’t change the sequence they’re processed in.

1 Like
3

Great, thank you!

Just one more question:

use Oban.Pro.Worker, chain: [by: [:worker, args: :organization_id]]

Is this the configuration you suggest or was it just copy&pasted from the docs?

I’m not sure whether having the :worker in there would achieve what I want. All workers, A, B, C are separate Oban.Pro.Worker modules, so creating the chain based on the :worker might not guarantee the sequence A -> B -> C, right?.

So, maybe their configuration should be:

use Oban.Pro.Worker, chain: [by: [args: :organization_id]]
4

It was copied and pasted then slightly modified. As they are three different workers, you’d probably want to omit that and just use the organization_id as you mentioned.

1 Like
5

Perfect! Thank you!