smaximov
Supervised GenServer process disappeared and wasn't restarted
Greetings!
Tonight we had an incident in production when a somewhat important GenServer process seemingly disappeared and had not been restarted by a supervisor. This is the first incident in 2.5 years since this worker had been introduced.
The GenServer in question has basically the following structure:
defmodule MyWorker do
use GenServer, restart: :transient
def start_link(opts) do
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
end
@impl GenServer
def init(opts) do
Process.send_after(self(), :do_work, 500)
{:ok, opts}
end
@impl GenServer
def handle_info(:do_work, state) do
do_some_work(state)
Process.send_after(self(), :do_work, 500)
{:noreply, state}
end
defp do_some_work(state) do
# query DB
# send HTTP request
# that's all - no OTP messaging, exits, etc.
end
end
This GenServer is started as part of a Supervisor (let’s call it MySupervisor). MySupervisor uses the :one_for_one strategy, all other init options are default.
When I connected to the remote console, I found out that the process is not alive - Process.whereis(MyWorker) returned nil, Supervisor.which_children(MySupervisor) returned the following entry for MyWorker:
{MyWorker, :undefined, :worker, [MyWorker]}
As far as I understand, this means MyWorker’s child spec was known to MySupervisor, but the worker process itself was not running. I restarted MyWorker with Supervisor.restart_child/2 and started figuring out what went wrong.
The obvious culprit is restart: :transient - as MyWorker was intended to work indefinitely and to never successfully terminate, it should have been :permanent. If we assume MyWorker terminated normally, it explains why it wasn’t restarted.
The problem is figuring out why MyWorker died, because it neither stops itself, nor it is explicitly stopped by some other app code. I have 2 hypotheses left:
- some developer connected to the remote console and stopped the worker by hand (very unlikely);
- the worker terminated with an error and was not restarted due to a bug in
Supervisor(extremely unlikely).
Am I missing anything else?
Most Liked
mudasobwa
There are two things to take into consideration as well:
① Restart strategies, specifically :max_restarts - the maximum number of restarts allowed in a time frame. Defaults to 3 and :max_seconds - the time frame in which :max_restarts applies. Defaults to 5. If something went south and there were 4 attempts to restart it within 5 seconds interval (defaults,) all failing, the GenServer would end up not restarted. This is likely your case, if e. g. the DB connection has dropped for 3 seconds. The culprit would be 500ms interval in init/1, which makes 4 consequtive insuccessful attempts to execute do_some_work/1 to fit into 5 secs and shutdown the process forever.
② The mailbox overflow. This technically might be your case, but extremely unlikely, just saying for the complete picture.
LostKobrakai
The app will terminate if the root pid exits, and immediatelly on the first exit, no restarts. And if the app is :permanent it’ll make the whole vm stop.
lud
Yes the supervisor will exit if it has to restart a child too many times. And it will be restarted by its own supervisor.
You can have a chain of Sup → Sup → Sup → Sup → Worker and all elements of the chain will behave like this.
A good way to think about this is that the code describing the supervisor tree (each supervisor children, and the children description in each child supervisor, and so on) describes “how the app state should be at runtime”. And the system will try its best to maintain that state, or otherwise fail and shutdown totally.
Of course that description can be altered at runtime by calling start_child, delete_child to add/remove parts of the tree, and of course having :transient, :temporary or :significant children.
But there is no way it will skip some parts of the tree unless you tell it to. Max-restarts and al’ will only have a temporary impact on the state of the system, but the app will always get back to the desired state.
So if my understanding is correct, if
MyWorkerwas linked to another process, and that process exited with a:shutdown(or{:shutdown, value}), then it would indeed causeMyWorkerto terminate normally as well. Unfortunately,MyWorkerisn’t linked to any other process.
If the child spec for MyWorker has restart: :transient then yes! (:shutdown is not “normally” but the supervisor will not restart it indeed).
Any progress on finding the cause?
- Another possibility is that the child spec is overriden somewhere and
:transientis replaced with:temporary; but I guess this is unlikely. - Are there any calls to
Supervisor.terminate_childorGenServer.stopin your codebase? - Are you using
:significantwith:any_significantor:all_significant?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"