fhunleth
Guidelines for Supervision trees and setting restart intensity parameters
I was wondering if there were any books, blogs, or docs out there that discussed supervisor restart intensity parameters (number of crashes allowed in a period before the Supervisor crashes) and how to handle top-level application supervisor crashes from exceeding these.
For example, there’s some information in Supervisor Behaviour — Erlang System Documentation v29.0.2. This is helpful, but the numbers seem arbitrarily picked (this may just be the nature of these numbers).
There also seems to be some unwritten(?) knowledge about handling top-level application supervisor crashes due to exceeding restart intensities. If you’re using a 3rd party application, it will have selected a restart intensity for you. If this doesn’t work for your project, I’ve heard of people starting the OTP application as temporary (instead of permanent) and adding code to monitor whether the application is still running and if not, to have a cooldown period before restarting it. shoehorn gets into this and other strategies, but it doesn’t provide recommendations.
There’s also the give up, let the Erlang VM crash, and have systemd restart it approach.
My hope is that a few people have gone pretty far down the above or similar paths and have written up their experiences. All pointers appreciated.
Most Liked
michaelkschmidt
This has been a major learning point for our project as it has matured. The whole point of limiting the supervisor restarts is to allow the next layer up to try to recover the system. For some things this is the right thing to do, but for others it is not.
For example, we were spawning an external program written by another team. No amount of restarts on our side will fix will help it recover, so we wound up using a “circuit breaker” (GitHub - jlouis/fuse: A Circuit Breaker for Erlang · GitHub). It allowed us to restart the program an infinite number of times, while still getting “cooldown” periods where we leave it off. This helps the system recover, as it is not spending all its cycles on restarting, and it also helps resolve random race conditions where the program could not start due to something not being ready.
The other problem we faced was the OTP :ssh application. During penetration testing, the app would simply die and take out the entire VM with it. We were able to use Shoehorn to keep restarting it, and things recovered nicely as soon as the attack subsided.
ConnorRigby
Similarly to @michaelkschmidt, i have an anecdotal experience with Supervision trees. This is my monolithic Nerves application, that has been steadily growing for the better part of two years:
I’ve found that when designing my tree, even simple gen_servers wrapping other’s libraries end up with their own special supervisor. For example the currently available sqlite3 adapter for ecto has a pretty nasty race condition in it that if left on the root supervisor (such as in a Phoenix app) will cause the app to fail to start. if you nest it a few levels deep or even nest it behind another OTP Application instead of just a normal supervisor, it is “stable enough”.
Side note: most (if not all) of my experience in this is in Nerves (as opposed to say Phoenix). I’ve found that managing supervision trees isn’t as much of a problem for (at least simple) Phoenix apps since you can make assumptions such as “the time will be set correctly by the time my app starts”
tty
There might even be a time where you find the best is to isolate the external library into a slave node and have it auto restarted by a supervisor/gen_server.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"