Alternatively, you can use my Parent library, which would allow you to have a sort of a smart supervisor. The benefit of a parent is that you don’t need a separate process to deal with restarts. You can have a sort of a crossover between a GenServer and a Supervisor in a single process.
I’m still not convinced that this is the right course, but I have to go out now, so I might post more comments/questions later 
This exactly. Supervision restart is the wrong thing to fall back on if your process can’t access a server for hours at a time. For one thing, if you have a large number of these processes, your supervisor becomes a substantial bottleneck for managing the connectivity of these processes. Instead each process should manage its own connection, disconnection, and reconnection life cycle. The supervisor will be there if something truly unexpected happens.
I feel there’s a misconception about what supervisors are supposed to do.
Supervisors try to keep an application in an available state by restarting crashing children and hoping that resetting the internal state of the failing process/(sub)system does fix things or at least that any external factors causing the crash were temporary hick-ups. If things don’t get better your application as a whole is considered to be unavailable – not doing its job – and therefore it’s stoped and hopefully restarted from the OS level, so maybe that level of restart can bring things back to working. If even that doesn’t help there should (in theory) be no difference between the application running, but continuously failing, and the application not running at all.
If a process/(sub)system (continually) failing is not considered critical enough to bring down the application as a whole then this needs to be handled by something outside the supervision tree, which can make different decisions about what to do in the event of failure.
The only reasons I see which would cause this are:
In my opinion all of these situations should be handled explicitly, instead of using supervisor restarts.
I created quite a few applications where GenServers under a DynamicSupervisor either hold state and push it somewhere over http or get state from somewhere and hold it.
As others already said, it’s better to catch any http errors and use a retry strategy while keeping the GenServer alive.
Thank you everyone for the comments and opinions. They truly helped me shape the understanding of how things are supposed to work in Elixir and which approach to take.
When I first entered the world of Elixir and Erlang, I was fascinated by the advertisement of being able to have an application running in production for 7 years in a row without a single day of downtime. My objective is to capitalize on that promise.
I guess that would make sense.
An advice that I will surely follow, now that I have read everything!
I am not familiar with circuit breakers. Can you point me to some literature on the matter ?
So, your recommendation of using the Parent library still holds? I like the idea that I can have a supervisor simply update a metric every time a child fails and that keeps trying to create children to remake the connections.
However from what I have read, I understand that Supervisors should be used only when things go wrong, therefore in this case perhaps using Parent would be against the community’s advise.
No. I would handle disconnect and failing connection attempts directly in the worker process without stopping the worker.
Or more precisely, only when things go unexpectedly wrong.
I discussed this in my talk at elixirconfeu. But if you’re dealing with connections to downstream services then you’re going to want to treat the connections as a state machine. They’ll come up in a disconnected state and attempt to connect over time. If they can connect then they move to the connected state and can start accepting traffic and sending requests. The connection processes can handle some number of known issues such as disconnects or 503s or something. But part of the benefit of treating your connections as a state machine is that allowing them to crash is a safer proposition. When the supervisor restarts the connection process the connection process won’t expect so it’s less likely to enter a crash/restart loop.
This is what I meant when I said you need to bring your processes up in a stable state. Processes that talk to the outside world should have no expectations that the downstream service is available when they start. Otherwise you’ll end with a system that can’t recover from transient failures and probably can’t even boot reliably.
This also gives you the observability that you want. It’s generally not helpful to page for every crashed process because you won’t have enough context to create a meaningful page. But if your processes are attempting to move from a disconnected state into a connected state and aren’t able to then you can start to send more meaningful alerts to an operator or an on-call person. You can also regularly poll your connection processes and if some number of them aren’t up then you can take yourself out of the load balancer or provide more meaningful liveness and readiness checks to your other systems.
Most services run in a conjunction with other services. The meaningful metric is typically not whether one service is healthy. It’s whether the entire system is healthy. Allowing the system to gracefully degrade becomes a key factor for healthy systems.