Running jobs only once in a cluster

I need to run an import job once every minute in the cluster, basically replicating the functionality of a Kubernetes job. I first tried to use Quantum core, but seems like the “singleton” functionality was removed.

How do you guys do it ?

You can take a look at the highlander library.

see this example https://github.com/connorlay/pulsar it’s not mine, i found it in a similar topic on this forum. (it’s using highlander and periodic (that’s part of the ‘parent’ library). Docs for periodic are here: https://hexdocs.pm/parent/Periodic.html#content

In this example highlander makes sure you have only a single process(job) running in your cluster, while periodic schedules the job every minute.

If it’s really important that it’s steady and you can use a (postgres) DB i would also advise to take a look at Oban

2 Likes