Same Chunk worker, different chunk sizes, timeouts, queues

Special Sections Oban
, ,
1

Continuing on the discussion from https://elixir-lang.slack.com/archives/CRDGH1XCM/p1715960902329219

I would like to use the same Chunk worker but have different chunk sizes and chunk timeouts depending on the context of the job. For example, imagine a chunk worker that chunks by channels: Email, Text, and Push. The worker is able to use some sort of polymorphism so that the worker code is fairly simple:

defmodule MyChunkWorker,
  use Oban.Pro.Workers.Chunk, by: worker

  def process(jobs) do
    jobs
    |> batch_messages()
    |> send()
  end
end

After some hacking around, I found that I can input my own meta like so:

  def test do
    jobs_a =
      for _ <- 1..10 do
        __MODULE__.new(%{queue: :test_a},
          queue: :test_a,
          meta: %{
            chunk_size: 10,
            chunk_timeout: 1000
          }
        )
      end

    jobs_b =
      for _ <- 1..10 do
        __MODULE__.new(%{queue: :test_b},
          queue: :test_b,
          meta: %{
            chunk_size: 5,
            chunk_timeout: 1000
          }
        )
      end

    jobs_c =
      for _ <- 1..10 do
        __MODULE__.new(%{queue: :test_c},
          queue: :test_c,
          meta: %{
            chunk_size: 50,
            chunk_timeout: 1000
          }
        )
      end

    Oban.insert_all(jobs_a ++ jobs_b ++ jobs_c)
  end

Would this be the blessed Oban way to configure chunk_size and timeouts? Also, if I only care about chunk_size and chunk_timeout, is there a reason to separate the different types of jobs into different queues?

Thanks!

2

Overriding the chunk_* meta is the correct way to configure chunks differently. It’s not explained in the docs because it’s a little clunky and the exact meta keys are considered as an implementation detail.

You don’t have to separate them into different queues, but you will need to chunk them by something distinctive. Otherwise, because you’ve chunked by: :worker, any one of those chunks could intermix in the same queue and you wouldn’t have the size or timeout you want.

An alternative, instead of chunking by worker, is to partition on args:

use Oban.Pro.Workers.Chunk, by: [args: :some_key]
1 Like
3

Thanks for confirming!

1 Like