Running quantum job from single node

aviraj

We have elixir application deployed on two servers. The application has been configured with some quantum jobs which execute from both servers at the same time. We want to limit execution of the quantum jobs to a single server. As per documentation, we could see the following options,

Add global?: true to the job configuration
run_strategy: {Quantum.RunStrategy.Random, :cluster}

We tried option (1), it didnt work. The job still executed from both servers. Do both (1) and (2) need to be set at the same time in a job? or there something else missing in the configuration

config :titan, Titan.Scheduler,
  timeout: 20_000,
  global?: true,
  jobs: [
    job_schedule: [
      schedule: "20 6 * * *",
      task: {Titan.ContentSchedule, :schedule_updates, []}
    ],
]

How does this clustering work? Even without giving the list of nodes, how do the two servers communicate among themselves to co-ordinate the job execution?

16 comments

#quantum #scheduled-tasks

1 4190 16

2021-04-15 18:16:00 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

hubertlepicki

Global has been removed in 3.0.0. quantum-core/CHANGELOG.md at 0ac628950b8de0a03ed914f1ca0c77c9ad93c4ac · quantum-elixir/quantum-core · GitHub
which is kind of soliving the problem because the implementation wasn’t great and was causing us a lot of trouble with unexpected behavior.

You probably want to use Oban, which will guarantee scheduled jobs uniqueness on the PostgreSQL level:
https://github.com/sorentwo/oban#periodic-jobs

You can also use something like GitHub - SchedEx/SchedEx: Simple scheduling for Elixir · GitHub but you will have to wrap it with own code to make only one process in cluster is started per job, I have been doing that with Horde.DynamicSupervisor. But honestly, if asked to implement it today, I would just use Oban.

Post #1

aviraj

Thanks. I forgot to mention, the project uses quantum 2.2
Right now, we dont have the option to go with another solution, we have to make this work with the current versions of libraries if possible.
Would putting a run strategy work?
run_strategy: {Quantum.RunStrategy.Random, :cluster}

Post #2

poops

I had this same issue where I had to move off Oban because it locked a table and took our entire application down. Moved to quantum 3 and used highlander to have quantum run on only one process:

https://github.com/quantum-elixir/quantum-core/issues/411#issuecomment-652733990

Post #3

hubertlepicki

ah, that’s quite the opposite what I did. I used Horde.DynamicSupervisor to distribute sched_ex processes across cluster. If I understand in your solution there’s always going to be single node that executes these scheduled tasks, all at the same noode.

Interesting to learn about oban locking the scheduled jobs table, I never experienced such issue.

Post #4

sorentwo

Oban Core Team

I would like to hear more about this as well. What version of Oban were you using? How many jobs were you scheduling?

Scheduling only uses an advisory lock, which isn’t associated with a table at all. I’m very curious!

Post #5

aviraj

Any suggestions on how to get the RunStrategy working so that the jobs run on a single node in the cluster? Somehow adding the RunStrategy as randon did not have any effect on the jobs. The jobs were still executing from both nodes

Post #6

poops

Correct, we didn’t need the crons to be distributed. Just needed to ensure only one process was running them.

Post #7

poops

Was using oban 2.0.0, oban_pro 0.3.0, and oban_web 2.0.0. It was a recent upgrade to those versions, with the following plugins:

 Oban.Plugins.Pruner,
 Oban.Pro.Plugins.Lifeline,
 Oban.Web.Plugins.Stat

Database CPU was pinned at 100% and our DBA said it was related to some queries to the oban_jobs table.
I don’t have exact numbers as the table eventually got pruned, but I believe there were hundreds of thousands of completed job records. My guess was that the pruning plugin locked things up, but I haven’t had a chance to look into it.

Post #8

connorlay

Hey! I recently solved this problem of running globally unique jobs in a clustered Elixir application. The project was written using Quantum 2, which had built-in support for clustering via the global: true mode. In our experience the implementation of global jobs was unreliable and we found that many jobs would stop executing entirely.

After doing some research, we ended up moving from Quantum to periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster.

I have a proof-of-concept here that shows the basic functionality.

Post #9