How does Oban Pro burst mode allocate slots across partitions and priorities?

Questions & Help Questions
,
1

How does Oban Pro burst mode allocate slots across partitions and priorities?

I’m trying to understand how burst: true behaves with partitioned global limits in Oban Pro.

For example, with a queue like:

queue1: [
  local_limit: 5,
  global_limit: [
    allowed: 1,
    burst: true,
    partition: [args: :organization_id]
  ]
]

My understanding is:

  • Each partition gets 1 guaranteed slot (allowed: 1)

  • With burst: true, partitions can temporarily use additional available slots up to the queue’s available concurrency (local_limit: 5 in this case)

However, I’m confused about how burst slots are actually distributed.

Scenario 1 — Two active partitions

If there are jobs from 2 partitions only, I expected burst mode to greedily utilize all available slots.

But instead of something like:

partition_a = 3
partition_b = 2

I’m seeing behavior closer to:

partition_a = 2
partition_b = 2
1 slot idle

Why is the remaining slot left unused?

Scenario 2 — Three active partitions

Similarly, with 3 partitions active, I expected burst to continue filling remaining capacity, but it seems closer to:

partition_a = 1
partition_b = 1
partition_c = 1
2 slots idle

instead of using the full queue capacity.

Does burst mode not greedily allocate all available slots?

Scenario 3 — Priority interaction

I also noticed that priority doesn’t seem to affect burst allocation the way I expected.

Suppose:

  • Partition A jobs have priority 0

  • Partition B jobs have priority 0

  • Partition C jobs have priority 1

I expected all burst capacity to be consumed by priority 0 jobs first before any priority 1 jobs ran.

Instead, it seems like slots are distributed across all partitions regardless of priority, e.g.:

A = 1
B = 1
C = 1

Why does burst allocation behave this way?

How exactly does Oban Pro decide:

  1. Whether additional burst slots should be used

  2. Which partition receives them

  3. Whether priorities are considered before burst distribution

  4. Why queue capacity may remain idle even when runnable jobs exist

2

It’s hard to answer definitively without knowing which Pro version you’re using, but I’ll answer based on how v1.7 works.

Burst here means “let each partition exceed the global allowed limit,” not “fill every slot”. The calculation favors fairness, so additional capacity isn’t given to a random partition. That’s why you’ll see a “2/2” split rather than “3/2”.

All currently-active partitions, where active is a briefly cached (3s) list of partitions with available jobs. Leftover capacity is discarded, not redistributed, in the interest of fairness.

Priorities are only honored within each partition’s query, it doesn’t apply priority between partitions. That’s semantically because of how the query and the underlying index are structured. Applying priority before the partitioning would require an additional pre-fetch query, which would double the work. A hybrid may be possible that gets closer to the priority behavior you expect.

This may be down to available partition caching. If there are jobs spread across multiple partitions, it may take several seconds before a job is in a visible partition.

3

Thanks, this clears up a lot of things.

Just one additional thing related to partitioning + priority, you mentioned that a hybrid approach may be possible. Could you explain a bit more about it? btw i have oban_pro 1.6.13

4

That’s a recent enough Pro version that all of the details from above apply.

There isn’t anything you can tweak to achieve a “hybrid” mode. But, after this discussion, the next v1.7 patch will include a change that better utilizes all available slots :slightly_smiling_face:

5

Great, will be waiting for it.