How to make a Supervisor **never** die?

The point of max_restarts (and max_seconds) option is to break the endless restart cycle, and move recovery to the higher level. Thus, in my opinion, an infinity restart option doesn’t make sense. You could approximate it by e.g. using some insanely large number for both options, but I’d advise against doing it.

There are some alternative approaches which could be considered, but first I’d like to learn more about what do these workers do, and why can they restart so frequently?

Totally agree with @sasajuric here. The supervisor’s job is to bring up your child processes in a stable state. If the supervisor can’t achieve that steady state then you want to continue restarting at a higher level until you eventually shut down the application (because at that point something has gone terribly wrong).

I think in this case you may want to rethink your problem. We don’t really have enough information to say but if your workers really are that transient then I would probably move the workers into a dynamic supervisor and then monitor and start them from some other controlling process. But there are a lot of options here. The important thing is to think about the problem in terms of what guarantees you want your supervisor to provide.

These workers use gun to open HTTP2 connections to other servers. On startup, each worker opens a connection to a given domain and then keeps that connection open. This allows us to send an insane amount of requests per second.

The issue here is that when the connection times out or dies, the worker is smart enough to try and reconnect. But what if it is impossible to reconnect? Maybe there is some corrupt state in the worker or maybe there is something else. In this scenario (and other simillar scenarios) I allow the worker to die so the supervisor can create a new one with a fresh state (as advised by your book )

But what happens when a worker suicides over and over again? The Supervisor will try to restart it over and over again until it decides this is a lost cause, and then suicides himself as well.

Workers will fail. The systems they connect to may be unavailable for hours or days at a row. If my supervisor crashes, it starts a chain reaction that causes the app to crash. Thus my approach is to make the Supervisor never die while still being able to allow workers to die so they can come back online with a clean state.

I don’t want this to happen. Ever.

You’re fixating on the wrong part of that statement.

^ thats the important part. You need to think about how to bring up your connection processes in a stable state and assume that they’ll eventually crash. If you aren’t bringing them up in a stable state to begin with then you’ve designed a fragile system.

If you have designed the system to handle the connections correctly and your supervisor still can’t bring up your system into a steady state then you absolutely want to surface that error. Eventually if this propagates far enough up then you’ll want to shutdown. If you’ve done your design well and you reach the point of shutting down then it means that something drastic has gone wrong outside of your control. To put it a different way, who cares if your service is up if it can’t do anything useful or always serves the wrong answers? Its functionally equivalent to being shutdown anyway.

Don’t design around promises you can’t keep. This is central to the erlang philosophy. Your server will go bad. The network will go bad. Design to handle these.

Yes, but I rather have a useless service online that keeps pinging the team with error messages rather than having it crash completely. This is a design choice.

Yes, but it doesn’t mean I have to completely shutdown the entire application. I rather have it online checking for service availability every chance it gets and posting error messages to some service constantly than having it quit and shutdown.

As you describe it this is expected, not exceptional behaviour. Design needs to accomodate expected behaviour.

Sounds like working against live servers and watching for dead ones to come back online are distinct responsibilities that need to be handled separately (i.e. when a worker decides that the server won’t respond it terminates normally handing matters back to the dead server watcher).

It sounds like you need to manage this with some form of backoff. You can’t code to prevent anything bad from ever happening, that’s just not possible. But if you have some idea of what can go wrong (and it sounds like you do), you should handle those expected errors. This might be in the form of try/catches, exponential backoff in trying to connect with gun etc. You would in this case put that logic in the worker, not in the Supervisor, and not rely on crashes to refresh the state in cases of known errors.

If that’s for some reason not possible, set the worker strategy to transient and let a separate process handle the restarts. This is not exactly the reason for, but kind of fits into, the use case for the new Registry.select in 1.9, where you would be able to register your workers and then query the Registry for the status of the workers, restarting as required. You’d then be able to keep track of workers that keep crashing, backoff and notify/alert, but keep the other workers living. This can also be implemented by using Process.monitor. Note that this is making your life difficult for yourself, and you’re introducing a lot more risk for corrupted states.

When it comes to the unknown, there’s no way for you to keep the system working in a functioning condition. For those cases crashing is correct.

You can protect your workers with circuit breakers, one straight forward way to do this is to use this tiny application/library GitHub - jlouis/fuse: A Circuit Breaker for Erlang · GitHub