Is there a good way to dequeueing jobs with unique constraints in Oban?

Questions / Help
1

There are unique constraints for enqueueing jobs in Oban, but not for dequeueing jobs.

Scenario:

I’m running a B2B SaaS that performs periodic tasks.
I want to run multiple jobs simultaneously, but one job for each organization.

For example,

defmodule MyWorker do
  use Oban.Worker,
    queue: :my_queue,
    unique_dequeue: [fields: [:queue, :worker, :args], keys: [:org_id], period: 60] # <= I want this option, not enqueueing uniqueness

end

enqueueing jobs

  1. %{org_id: 1, report_id: 1}
  2. %{org_id: 1, report_id: 2}
  3. %{org_id: 2, report_id: 3}

All jobs are scheduled at the same time.

dequeueing job

  1. perform (org_id: 1, report_id: 1), (org_id: 2, report_id: 3) at once
  2. perform (org_id: 1, report_id: 2) after (org_id: 1, report_id: 1) is finished.

Is it possible in Oban?

2

That is possible with global partitioning available in Pro’s SmartEngine. You’d set a global limit of 1 and partition by org_id, like this:

queues: [
  my_queue: [local_limit: 10, global_limit: [allowed: 1, partition: [fields: [:args], keys: :org_id]]]
]

That allows 10 concurrent jobs per node for different `org_id` values.
2 Likes