Hello @armanm!

In general the idea is:

You should always have a strategy how to restart a process (think of if it crashes on the same node and gets restarted instantly by its parent supervisor.. for example, store in the DB some key part of the state that could help to reinitialize the process from the step it was terminated, but don’t blindly try to store the whole state).

Now when we know that we can start our process, stop it and continue with the same initial arguments, there are 2 general ways to provide with zero down time deployments (at least I’m familiar with):

1. Using distributed process registries (for example built in :global) and distributed supervisors.
   I’d recommend to look at Pogo. There is an article about it from @szajbus Pogo - distributed supervisor for Elixir.

   There are other options like swarm, Horde, syn you could take a look at. They all have their pros and cons.

   But one thing is common: they are eventually consistent!
   That might be fine for some use cases (when it’s ok to have a duplicate process for short period of a time when network split happen), but might not be acceptable for other scenarios. (There are companies that even have an “internal ban” on those libraries because some new devs often would think of them as an easy solution for a “single global process” and get burned by their eventually consistent nature).

   I’d also recommend to read The dangers of the Single Global Process

   There are implementations of strongly consistent consensus available for erlang and elixir (e.g. ra, and waraft) but I’m not aware of any distributed process registry or a supervisor implementation based on those.

2. Rely on 3rd party “queue”.
   For example, RabbitMQ. There is even an official tutorial with Elixir RabbitMQ tutorial - Work Queues | RabbitMQ
   In a nutshell:

   1. You place the job to the queue.
   2. One of the workers picks up the job and starts the long running process.
   3. Only once the process is done - worker acks the job.

   If the connection get’s closed because of the node termination - job get’s requeued and other active worker picks it up and starts the process (NOW here, you want to make sure that when process is restarted, it “continues” the work as mentioned earlier!)

   Another option could be “holding” a DB transaction lock. (I think that’s what Oban uses for its queues). But that might be more limiting than RabbitMQ.

   I would say, relying on 3rd party (single source of truth) queue-like mechanism is the more battle tested and trusted approach.

Agree, this is the most important part. It is very important to break down a big task into smaller tasks, that can be retried once the node has been restarted, without a big time loss in the overall execution of the task.

As for the persistence of jobs, nowadays I always reach for Oban, as not only you most probably will always have a DB laying around the server, but it solves 95% of potential problems you might encounter down the road out of the box.

Some inspirational viewing on the topic: