aviraj
We have elixir application deployed on two servers. The application has been configured with some quantum jobs which execute from both servers at the same time. We want to limit execution of the quantum jobs to a single server. As per documentation, we could see the following options,
- Add
global?: trueto the job configuration run_strategy: {Quantum.RunStrategy.Random, :cluster}
We tried option (1), it didnt work. The job still executed from both servers. Do both (1) and (2) need to be set at the same time in a job? or there something else missing in the configuration
config :titan, Titan.Scheduler,
timeout: 20_000,
global?: true,
jobs: [
job_schedule: [
schedule: "20 6 * * *",
task: {Titan.ContentSchedule, :schedule_updates, []}
],
]
How does this clustering work? Even without giving the list of nodes, how do the two servers communicate among themselves to co-ordinate the job execution?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Master modern tools, write cleaner code, and tackle real-world challenges with Manning books, videos, and courses. Start building today.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 16 to 7- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
bartblast
There is also the Citrine package, that aims to solve the cron clustering problem.
poops
Sure, there were 25 queues running. I also ran
SELECT * FROM oban_jobs_id_seqto see the last ID used, and it was 2915796. So we had close to 3 million jobs, although I don’t know how many were in there at the time of the spike.Ah ok. I assumed it was the pruner because it happened right after enabling it. It could’ve been a coincidence. These were the only three plugins running at the time:
sorentwo
Those are the primary queries used to stage scheduled jobs and to fetch them for execution. They’re fully indexed and should be very fast, < 1ms under normal load. How many queues are/were you running? I’d love to know more to help you all, or other people in a similar situation.
It would be configured in the
pluginssection of your config, using the Dynamic Pruning Plugin. That is all moot though, since the queries you shared aren’t from pruning.poops
Here are the 2 queries our DBA sent me:
UPDATE “public”.“oban_jobs” AS o0 SET “state” = $1 WHERE (o0.“id” IN (SELECT so0.“id” AS “id” FROM “public”.“oban_jobs” AS so0 WHERE (so0.“state” IN (?,?)) AND (so0.“queue” = $2) AND (so0.“scheduled_at” <= $3) FOR UPDATE SKIP LOCKED))UPDATE “public”.“oban_jobs” AS o0 SET “state” = $1, “attempted_at” = $2, “attempted_by” = $3, “attempt” = o0.“attempt” + $4 WHERE (o0.“id” IN (SELECT so0.“id” AS “id” FROM “public”.“oban_jobs” AS so0 WHERE (so0.“state” = ?) AND (so0.“queue” = $5) ORDER BY so0.“priority”, so0.“scheduled_at”, so0.“id” LIMIT $6 FOR UPDATE SKIP LOCKED)) RETURNING o0.“id”, o0.“state”, o0.“queue”, o0.“worker”, o0.“args”, o0.“errors”, o0.“tags”, o0.“attempt”, o0.“attempted_by”, o0.“max_attempts”, o0.“priority”, o0.“attI’m not sure, I didn’t originally set it up. Where would that be configured?
sorentwo
I haven’t had any other reports of that, I would like to have seen the queries.
That is entirely normal. Even the demo at https://getoban.pro/oban has 600,000 completed jobs sitting around.
Any idea if you were using per-worker or per-state dynamic pruning? The pruner deletes 10k records every one minute by default, which is extremely fast.
connorlay
Fixed! Thanks for pointing that out
Exadra37
It’s a 404 link, maybe the repo is private?
connorlay
Hey! I recently solved this problem of running globally unique jobs in a clustered Elixir application. The project was written using Quantum 2, which had built-in support for clustering via the
global: truemode. In our experience the implementation of global jobs was unreliable and we found that many jobs would stop executing entirely.After doing some research, we ended up moving from Quantum to periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster.
I have a proof-of-concept here that shows the basic functionality.
poops
Was using oban 2.0.0, oban_pro 0.3.0, and oban_web 2.0.0. It was a recent upgrade to those versions, with the following plugins:
Database CPU was pinned at 100% and our DBA said it was related to some queries to the
oban_jobstable.I don’t have exact numbers as the table eventually got pruned, but I believe there were hundreds of thousands of completed job records. My guess was that the pruning plugin locked things up, but I haven’t had a chance to look into it.
poops
Correct, we didn’t need the crons to be distributed. Just needed to ensure only one process was running them.