aviraj

aviraj

We have elixir application deployed on two servers. The application has been configured with some quantum jobs which execute from both servers at the same time. We want to limit execution of the quantum jobs to a single server. As per documentation, we could see the following options,

  1. Add global?: true to the job configuration
  2. run_strategy: {Quantum.RunStrategy.Random, :cluster}

We tried option (1), it didnt work. The job still executed from both servers. Do both (1) and (2) need to be set at the same time in a job? or there something else missing in the configuration

config :titan, Titan.Scheduler,
  timeout: 20_000,
  global?: true,
  jobs: [
    job_schedule: [
      schedule: "20 6 * * *",
      task: {Titan.ContentSchedule, :schedule_updates, []}
    ],
]

How does this clustering work? Even without giving the list of nodes, how do the two servers communicate among themselves to co-ordinate the job execution?

Showing Posts 1 to 10

hubertlepicki

hubertlepicki

Global has been removed in 3.0.0. quantum-core/CHANGELOG.md at 0ac628950b8de0a03ed914f1ca0c77c9ad93c4ac · quantum-elixir/quantum-core · GitHub
which is kind of soliving the problem because the implementation wasn’t great and was causing us a lot of trouble with unexpected behavior.

You probably want to use Oban, which will guarantee scheduled jobs uniqueness on the PostgreSQL level:
https://github.com/sorentwo/oban#periodic-jobs

You can also use something like GitHub - SchedEx/SchedEx: Simple scheduling for Elixir · GitHub but you will have to wrap it with own code to make only one process in cluster is started per job, I have been doing that with Horde.DynamicSupervisor. But honestly, if asked to implement it today, I would just use Oban.

aviraj

aviraj OP

Thanks. I forgot to mention, the project uses quantum 2.2
Right now, we dont have the option to go with another solution, we have to make this work with the current versions of libraries if possible.
Would putting a run strategy work?
run_strategy: {Quantum.RunStrategy.Random, :cluster}

poops

poops

I had this same issue where I had to move off Oban because it locked a table and took our entire application down. Moved to quantum 3 and used highlander to have quantum run on only one process:

https://github.com/quantum-elixir/quantum-core/issues/411#issuecomment-652733990

hubertlepicki

hubertlepicki

ah, that’s quite the opposite what I did. I used Horde.DynamicSupervisor to distribute sched_ex processes across cluster. If I understand in your solution there’s always going to be single node that executes these scheduled tasks, all at the same noode.

Interesting to learn about oban locking the scheduled jobs table, I never experienced such issue.

sorentwo

sorentwo

Oban Core Team

I would like to hear more about this as well. What version of Oban were you using? How many jobs were you scheduling?

Scheduling only uses an advisory lock, which isn’t associated with a table at all. I’m very curious!

aviraj

aviraj OP

Any suggestions on how to get the RunStrategy working so that the jobs run on a single node in the cluster? Somehow adding the RunStrategy as randon did not have any effect on the jobs. The jobs were still executing from both nodes

poops

poops

Correct, we didn’t need the crons to be distributed. Just needed to ensure only one process was running them.

poops

poops

Was using oban 2.0.0, oban_pro 0.3.0, and oban_web 2.0.0. It was a recent upgrade to those versions, with the following plugins:

 Oban.Plugins.Pruner,
 Oban.Pro.Plugins.Lifeline,
 Oban.Web.Plugins.Stat

Database CPU was pinned at 100% and our DBA said it was related to some queries to the oban_jobs table.
I don’t have exact numbers as the table eventually got pruned, but I believe there were hundreds of thousands of completed job records. My guess was that the pruning plugin locked things up, but I haven’t had a chance to look into it.

connorlay

connorlay

Hey! I recently solved this problem of running globally unique jobs in a clustered Elixir application. The project was written using Quantum 2, which had built-in support for clustering via the global: true mode. In our experience the implementation of global jobs was unreliable and we found that many jobs would stop executing entirely.

After doing some research, we ended up moving from Quantum to periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster.

I have a proof-of-concept here that shows the basic functionality.

Exadra37

Exadra37

It’s a 404 link, maybe the repo is private?

Where Next? Top

Trending in Questions Top

itzmidinesh
Hi all, I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
saleyn
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New
rlipscombe
When I use Mix.install with a git dependency, using git@github.com:... how do I change the SSH identity used? I’ve got two Github accoun...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews