I need to run an import job once every minute in the cluster, basically replicating the functionality of a Kubernetes job. I first tried to use Quantum core, but seems like the “singleton” functionality was removed.
How do you guys do it ?
You can take a look at the highlander library.
see this example https://github.com/connorlay/pulsar it’s not mine, i found it in a similar topic on this forum. (it’s using highlander and periodic (that’s part of the ‘parent’ library). Docs for periodic are here: https://hexdocs.pm/parent/Periodic.html#content
In this example highlander makes sure you have only a single process(job) running in your cluster, while periodic schedules the job every minute.
If it’s really important that it’s steady and you can use a (postgres) DB i would also advise to take a look at Oban
2 Likes
