aviraj

aviraj

Running quantum job from single node

We have elixir application deployed on two servers. The application has been configured with some quantum jobs which execute from both servers at the same time. We want to limit execution of the quantum jobs to a single server. As per documentation, we could see the following options,

  1. Add global?: true to the job configuration
  2. run_strategy: {Quantum.RunStrategy.Random, :cluster}

We tried option (1), it didnt work. The job still executed from both servers. Do both (1) and (2) need to be set at the same time in a job? or there something else missing in the configuration

config :titan, Titan.Scheduler,
  timeout: 20_000,
  global?: true,
  jobs: [
    job_schedule: [
      schedule: "20 6 * * *",
      task: {Titan.ContentSchedule, :schedule_updates, []}
    ],
]

How does this clustering work? Even without giving the list of nodes, how do the two servers communicate among themselves to co-ordinate the job execution?

First Post!

hubertlepicki

hubertlepicki

Global has been removed in 3.0.0. quantum-core/CHANGELOG.md at 0ac628950b8de0a03ed914f1ca0c77c9ad93c4ac · quantum-elixir/quantum-core · GitHub
which is kind of soliving the problem because the implementation wasn’t great and was causing us a lot of trouble with unexpected behavior.

You probably want to use Oban, which will guarantee scheduled jobs uniqueness on the PostgreSQL level:
https://github.com/sorentwo/oban#periodic-jobs

You can also use something like GitHub - SchedEx/SchedEx: Simple scheduling for Elixir · GitHub but you will have to wrap it with own code to make only one process in cluster is started per job, I have been doing that with Horde.DynamicSupervisor. But honestly, if asked to implement it today, I would just use Oban.

Most Liked

connorlay

connorlay

Hey! I recently solved this problem of running globally unique jobs in a clustered Elixir application. The project was written using Quantum 2, which had built-in support for clustering via the global: true mode. In our experience the implementation of global jobs was unreliable and we found that many jobs would stop executing entirely.

After doing some research, we ended up moving from Quantum to periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster.

I have a proof-of-concept here that shows the basic functionality.

poops

poops

I had this same issue where I had to move off Oban because it locked a table and took our entire application down. Moved to quantum 3 and used highlander to have quantum run on only one process:

https://github.com/quantum-elixir/quantum-core/issues/411#issuecomment-652733990

connorlay

connorlay

Fixed! Thanks for pointing that out :smile:

Last Post!

bartblast

bartblast

Creator of Hologram

There is also the Citrine package, that aims to solve the cron clustering problem.

Where Next?

Popular in Questions Top

baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54092 488
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New