Oban setup for separate worker/web nodes

If you’re using free version of Oban, it only has limit: local_limit option, i.e. you have local limit of 2 concurrent jobs in the default queue per worker. So, if you have 6 workers, you will have 12 concurrent jobs, if you have 12 workers you have 24… etc.

If you want to use global_limit, you will probably have to fetch Oban.Pro and use it’s Smart Engine (docs here: Smart Engine — Oban v2.11.0)

Hi @hubertlepicki,

I’m aware that the concurrency is a local configuration option. The issue I’m seeing is that when I insert 20 jobs using Oban.insert_all in my setup with 3 worker nodes and a queue with a concurrency of 1 results in 1 node starting all the jobs at once. What I’m expecting to see is 3 active jobs at once.

are you setting up those jobs as unique jobs?
per Oban docs:

The built-in Basic engine doesn’t support unique inserts for insert_all and you must use insert/3 for per-job unique support. Alternatively, the SmartEngine in Oban Pro supports bulk unique jobs and automatic batching.

Hi @cevado

Thank you for your reply. We are not using the unique jobs feature here.

The worker is initialised with:
use Oban.Worker, priority: 3, max_attempts: 1, queue: :ocr

The jobs are inserted with:
Oban.insert_all

And the queue is configured with:
queues: [default: 2, images: 1, ocr: 1]

Option 1 won’t work because you need the Oban supervisor for insert_all to work. The second option, where you disable those queues on your other nodes, is the proper way to do it. The Splitting Queues Between Nodes guide describes a solution for this exact situation, and Pro’s DynamicQueues plugin adds other conveniences.

That doesn’t sound right. Queues on a single node won’t run jobs beyond the concurrency limit. However, if they’re running fast enough, the other nodes may be unable to pick up the jobs. You can check where each job ran in the job’s attempted_by field and the exact time they started/finished at with the attempted_at and completed_at timestamps.

That wouldn’t make any difference. Uniqueness only effects inserting jobs; it has no bearing on how jobs are executed.

Thank you for replying! I’ve used the guide you’ve mentioned to configure the system.

The job takes a few seconds to finish. I’ve copied the data from the oban_jobs table a few days ago (not the best visualisation, null’s are not visible:

id queue worker  args  errors  attempt max_attempts  inserted_at scheduled_at  attempted_at  completed_at  attempted_by  discarded_at  priority  tags  meta  cancelled_at  state
20556197	ocr	Project.Ocr.OcrPageJob	{"page_id": "8e1bf275-00bc-4154-8fdd-f8c9a5641775"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556194	ocr	Project.Ocr.OcrPageJob	{"page_id": "e77e5a0b-eef6-4417-9137-e984284d732f"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556193	ocr	Project.Ocr.OcrPageJob	{"page_id": "d6e7f709-7fba-465f-b55b-b5707e9c5367"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556192	ocr	Project.Ocr.OcrPageJob	{"page_id": "a4217fc6-83a2-4edb-ae98-355e1d0e687a"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556190	ocr	Project.Ocr.OcrPageJob	{"page_id": "8f5d1815-7467-4528-9267-36c78dc91abf"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556189	ocr	Project.Ocr.OcrPageJob	{"page_id": "5fb1aa2b-de56-4169-a8d7-3877c0a93dd2"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing
20556187	ocr	Project.Ocr.OcrPageJob	{"page_id": "36a4b116-1a54-47d5-8c7d-01cbc6a0ecba"}	{}	1	1	2022-09-08 09:10:25.801283	2022-09-08 09:10:25.801283	2022-09-08 09:10:27.350033		{node@10.0.4.25}		3	{}	{}		executing

It stands out that all attempted_at values have the same timestamp.

Not sure if this is relevant, but in this project we have a cluster, so the nodes can talk to each other, but Oban is configured using the default LISTEN/NOTIFY PostgreSQL functions.

Config of the web process by running Oban.config():

%Oban.Config{
  dispatch_cooldown: 5,
  engine: Oban.Queue.BasicEngine,
  get_dynamic_repo: nil,
  log: false,
  name: Oban,
  node: "node@10.0.3.22",
  notifier: Oban.Notifiers.Postgres,
  peer: Oban.Peer,
  plugins: [
    {Oban.Plugins.Cron,
     [
       timezone: "Europe/Amsterdam",
       crontab: [
         ...
       ]
     ]},
    Oban.Plugins.Pruner,
    Oban.Plugins.Stager
  ],
  prefix: "public",
  queues: [],
  repo: Project.Repo,
  shutdown_grace_period: 15000
}

and the worker:

%Oban.Config{
  dispatch_cooldown: 5,
  engine: Oban.Queue.BasicEngine,
  get_dynamic_repo: nil,
  log: false,
  name: Oban,
  node: "node@10.0.4.27",
  notifier: Oban.Notifiers.Postgres,
  peer: Oban.Peer,
  plugins: [
    {Oban.Plugins.Cron,
     [
       timezone: "Europe/Amsterdam",
       crontab: [
         ...
       ]
     ]},
    Oban.Plugins.Pruner,
    Oban.Plugins.Stager
  ],
  prefix: "public",
  queues: [default: [limit: 2], images: [limit: 1], ocr: [limit: 1]],
  repo: Project.Repo,
  shutdown_grace_period: 15000
}

It certainly does. Which version of Oban are you running?

No, that doesn’t have any bearing. Notifications are used for pausing, canceling jobs, gossipping, etc., not for execution.

The version is 2.11.3.

I’ve never seen or heard of a queue ignoring the concurrency limit that way. You may be experiencing a bug fixed in Oban v2.12.1 due to subquery instability during a select for update.