Can I partition Oban queues by values of args, similar to the Unique feature?

I’d like a queue called :foo which processes Foo jobs. Foo jobs have a key argument. I’d like jobs with the same value of key to execute one after another, but if two Foos have different key, they can execute at the same time.

This sounds really similar to the Unique jobs feature, but to my best understanding, Unique isn’t quite it because if there’s a Foo with key “A” in the queue, and another Foo “A” attempts to enqueue, it won’t enqueue. That’s a roundabout way of saying I want to be able to enqueue as many Foo "A"s as possible, but only one can execute at a time.

https://getoban.pro/docs/pro/1.5.0-rc.7/Oban.Pro.Engines.Smart.html#module-queue-partitioning

Queue Partitioning in Oban.Pro.Engines.Smart sounds relevant, but the example isn’t complete. Something like this?

config :my_app, Oban,
  queues: [foo: [
    concurrency: 1,
    partition: [args: [:key]]
  ]],
  repo: MyApp.Repo

Global partitioning is the right track. Here are a couple some tweaks to make your version correct:

config :my_app, Oban,
  queues: [
    foo: [global_limit: [allowed: 1, partition: [args: [:key]]]]
  ],
  repo: MyApp.Repo

A smidge of a difference with allowed rather than concurrency, and specifying that it is global_limit.

1 Like

Thanks!

1 Like