Help with using Oban Pro Chunk workers

Special Sections Oban
, ,
1

I’m trying to understand how to use Oban Pro Chunk workers. I have a SendWorker that gets inserted thousands at a time. The SendWorkers get inserted with an “id” arg. I want to use Chunk workers so that I can use Chunk Partitioning and Queue Partitioning.

I have converted the SendWorker to a chunk worker, but when I insert multiple of these new SendWorkers, they still run independently of each other. Below is the sample code, but the log always displays the following no matter how many workers I insert.

SendWorker Chunk Starting with 1 Oban jobs

use Oban.Pro.Workers.Chunk,
    queue: :messaging,
    size: 1000,
    timeout: 5000

@impl true
def process([_ | _] = oban_jobs) do
   Logger.debug(["SendWorker Chunk Starting with #{length(oban_jobs)} Oban jobs"])

  ...
end

I’m sure I’m misunderstanding something or doing something wrong, can someone please help me understand how this should work?

1 Like
2

https://oban.pro/docs/pro/1.5.0-rc.4/Oban.Pro.Workers.Chunk.html#module-optimizing-chunks

Do you have a global queue limit? You’ll have one chunk, per concurrent job. Unless you have a great deal of jobs running, you typically want a single concurrent job:

queues: [
  chunked_queue: [global_limit: 1]
]
3 Likes
3

This was it, reducing my global limit resulted in the expected chunk size

4

Glad you got there!

P.S. I’m pretty sure @sorenone’s answer was the solution :smile:

1 Like
5

lol, fixed

6

My hero. :crossed_swords: :heart_on_fire:

1 Like