tompesman
Oban setup for separate worker/web nodes
Hi,
I’m tracing down an issue with Oban. I’ve a staging environment which works correctly and a production environment which has issues. When the Oban.insert_all is used to insert jobs to a queue all jobs are started at once while that specific queue is configured to have a concurrency of 1. One of the differences between staging and production is that there are multiple worker nodes. This gave me doubts about our queue configuration. On the web nodes we have the the queues set to: queues: [] and on the worker node set to: queues: [default: 2, images: 1, ocr: 1].
What is the correct configuration of Oban with a worker/web nodes setup?
- Start Oban in the
application.exonly on the worker nodes and you can still useOban.insert_allon web nodes - Start Oban on all nodes (web and worker), but with a different
queuesconfiguration
cc @sorentwo
Thanks
Most Liked
sorentwo
Option 1 won’t work because you need the Oban supervisor for insert_all to work. The second option, where you disable those queues on your other nodes, is the proper way to do it. The Splitting Queues Between Nodes guide describes a solution for this exact situation, and Pro’s DynamicQueues plugin adds other conveniences.
That doesn’t sound right. Queues on a single node won’t run jobs beyond the concurrency limit. However, if they’re running fast enough, the other nodes may be unable to pick up the jobs. You can check where each job ran in the job’s attempted_by field and the exact time they started/finished at with the attempted_at and completed_at timestamps.
That wouldn’t make any difference. Uniqueness only effects inserting jobs; it has no bearing on how jobs are executed.
hubertlepicki
If you’re using free version of Oban, it only has limit: local_limit option, i.e. you have local limit of 2 concurrent jobs in the default queue per worker. So, if you have 6 workers, you will have 12 concurrent jobs, if you have 12 workers you have 24… etc.
If you want to use global_limit, you will probably have to fetch Oban.Pro and use it’s Smart Engine (docs here: Smart Engine — Oban v2.11.0)
tompesman
Hi @hubertlepicki,
I’m aware that the concurrency is a local configuration option. The issue I’m seeing is that when I insert 20 jobs using Oban.insert_all in my setup with 3 worker nodes and a queue with a concurrency of 1 results in 1 node starting all the jobs at once. What I’m expecting to see is 3 active jobs at once.
Last Post!
sorentwo
You could hook into the queue producer’s telemetry events and emit logs:
defmodule ProducerLogger do
require Logger
def handle(_event, _measure, %{queue: queue, dispatched_count: count}, _opts) do
Logger.debug("[ProducerLogger] queue #{queue} dispatched #{count} jobs")
end
end
:telemetry.attach("queue-logger", [:oban, :producer, :stop], &ProducerLogger.handle/4, [])
The count should always be 1 based on your config. If not, something is awry.
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"