Oban Queues - Concurrent Jobs

queues: [mailers: 1] means there is one underlying process to process the queue, which means there is at most one job being executed at a time on a node. It’s unrelated to when and how many jobs are scheduled at a point in time. The worker would take a job from the queue, run it, and then if there are other jobs that are due, run them in order afterwards. That means that some jobs might be executed later than they are scheduled.

I should’ve read the question in full before attempting to answer.

Here’s a script that might be used to verify your theories.

queue.ex

# adapting https://github.com/wojtekmach/mix_install_examples/blob/main/oban.exs
# to make the job unsuccessful
Mix.install([
  {:ecto_sql, "~> 3.6.2"},
  {:postgrex, "~> 0.15.0"},
  {:oban, "~> 2.8"}
])

Application.put_env(:myapp, Repo, database: "mix_install_oban")

defmodule Repo do
  use Ecto.Repo,
    adapter: Ecto.Adapters.Postgres,
    otp_app: :myapp
end

defmodule Migration0 do
  use Ecto.Migration

  def change do
    Oban.Migrations.up()
  end
end

defmodule Main do
  def main do
    children = [
      Repo,
      {Oban, repo: Repo, plugins: [Oban.Plugins.Pruner], queues: [default: 1]} # 1 process queue
    ]

    Repo.__adapter__().storage_down(Repo.config())
    Repo.__adapter__().storage_up(Repo.config())
    {:ok, _} = Supervisor.start_link(children, strategy: :one_for_one)

    Ecto.Migrator.run(Repo, [{0, Migration0}], :up, all: true)

    Oban.insert!(Worker.new(%{id: 1}))
    Oban.insert!(Worker.new(%{id: 2}))

    Oban.Job
    |> Repo.all()
  end
end

defmodule Worker do
  use Oban.Worker
  require Logger

  @impl true
  def perform(%Oban.Job{} = job) do
    Logger.info("running job #{job.id}")
    1 / 0
  end
end

Main.main()
:timer.sleep(:infinity)

> elixir queue.ex
17:15:00.848 [info]  running job 1
17:15:00.862 [info]  running job 2
17:15:17.845 [info]  running job 1
17:15:18.847 [info]  running job 2
17:15:36.929 [info]  running job 1
17:15:37.935 [info]  running job 2
17:16:01.038 [info]  running job 1
17:16:02.042 [info]  running job 2
17:16:30.146 [info]  running job 1
17:16:33.157 [info]  running job 2

So Theory B doesn’t seem to hold as oban doesn’t get stuck on a single job.

The number in mailers: 1 is a concurrency limit. It regulates how many jobs may execute at once (concurrently) within that queue for that node.

Neither theory A nor B is quite right. A few notes to clarify:

• Scheduling is purely a timestamp. Once the timestamp is current the job is made available to run.
• Job execution is interdependent, meaning the failure of one job doesn’t block another job from running. In your scenario, if the first job fails then it is scheduled for a retry and the next available job may run.
• Concurrency limits are local to the node. If you run two nodes, or even do blue/green deploys, then you’ll have an effective concurrency of 2 instead of 1.

Thanks, that’s very helpful!