I’ve been evaluating Oban DynamicPrioritizer for a project, but it’s not totally clear how it works. I’ve read the docs and ran some proof of concepts, but it’s still unclear, can you please help me?
Does it always update 5 jobs in available state for more than 5 minutes to priority 0, no matter what’s the current priority, and decreases the remaining jobs priority by 1?
I was running a PoC where I inserted 10 jobs, one for each priority, from 0 to 9, with a Process.sleep/1 to keep the job running, and after 5 minutes it updates 5 jobs to priority 0 and then decreased the other jobs priority by 1. While the jobs were processing, I inserted a new job with priority 1 and it went to the end of the line, even with jobs lower priority than it in the same queue, because the plugin was always decreasing the priority of the others, it was only processed when its own priority was decreased to 0 too. So, I’m not sure if I got it right or its behavior varies.
I’d like to understand the plugin 100% in a way I can predict what it’s going to do, so it can reliable for our use case.
The DynamicPrioritizer bumps the priority of available jobs by 1 after the configured period, 5 minutes by default. Bumping the priority means decreasing it by 1 until eventually the priority is 0, or the job runs.
Only available jobs are changed. Running jobs have an executing state, and they’re not touched by the prioritizer.
Jobs with a higher priority (lower number) will all run first. If the queue’s concurrency is low, and the jobs process slowly enough, then it could easily block non-zero priority jobs because they wouldn’t have any space to run.
Got it, thanks. So if a job with priority 5 remains in available state for more than 5 minutes, it will be bumped to 4, then to 3, until it is bumped to 0, or is executed, minute by minute after the first 5 minutes.
That information about the bump by 1 is what’s missing in the documentation, it was not clear if it goes from whatever to 0 or by some number.
I think we won’t be able to use it, the configuration would need to be more granular. We have a queue with 4 workers with different priorities (0, 1, 3, 4). We want the workers with priority 4 eventually execute, bumping it to 2, but never to 1 or 0, so they can cut the line and execute before the jobs with priority 3. Other feature that we miss is how many jobs should be prioritized on every round, so only some jobs cut the line and the jobs with priority 3 still execute. We have this design to leverage the global limit by job args, all workers share the same partition key.
