Multiple instances of Quantum Worker running simultaneously resulting in duplicates

Questions / Help
, ,
1

fly.io multiple instance phoenix app for executing Quantum job scheduler but each instance machine execute quantum job. The Quantum process is running in duplicate, causing results to be stored redundantly in the database.

How can we ensure that the Quantum job process runs uniquely and does not execute in duplicate across multiple instances?

Please share any good ideas. Thank you.

2

We are facing a similar issue. Until now we have not really given it much though, since we use fly.io for all our feature branch testing, thus the redundancy was not critical. But now, while developing a new feature, it really bugs me.

Any insights or ideas would be most useful :slight_smile:

3

You start quantum in your process tree. So you can use any tooling, which helps have a single process per cluster instead of per node to accomplish this.