smaximov
Supervised GenServer process disappeared and wasn't restarted
Greetings!
Tonight we had an incident in production when a somewhat important GenServer process seemingly disappeared and had not been restarted by a supervisor. This is the first incident in 2.5 years since this worker had been introduced.
The GenServer in question has basically the following structure:
defmodule MyWorker do
use GenServer, restart: :transient
def start_link(opts) do
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
end
@impl GenServer
def init(opts) do
Process.send_after(self(), :do_work, 500)
{:ok, opts}
end
@impl GenServer
def handle_info(:do_work, state) do
do_some_work(state)
Process.send_after(self(), :do_work, 500)
{:noreply, state}
end
defp do_some_work(state) do
# query DB
# send HTTP request
# that's all - no OTP messaging, exits, etc.
end
end
This GenServer is started as part of a Supervisor (let’s call it MySupervisor). MySupervisor uses the :one_for_one strategy, all other init options are default.
When I connected to the remote console, I found out that the process is not alive - Process.whereis(MyWorker) returned nil, Supervisor.which_children(MySupervisor) returned the following entry for MyWorker:
{MyWorker, :undefined, :worker, [MyWorker]}
As far as I understand, this means MyWorker’s child spec was known to MySupervisor, but the worker process itself was not running. I restarted MyWorker with Supervisor.restart_child/2 and started figuring out what went wrong.
The obvious culprit is restart: :transient - as MyWorker was intended to work indefinitely and to never successfully terminate, it should have been :permanent. If we assume MyWorker terminated normally, it explains why it wasn’t restarted.
The problem is figuring out why MyWorker died, because it neither stops itself, nor it is explicitly stopped by some other app code. I have 2 hypotheses left:
- some developer connected to the remote console and stopped the worker by hand (very unlikely);
- the worker terminated with an error and was not restarted due to a bug in
Supervisor(extremely unlikely).
Am I missing anything else?
First Post!
lud
Hi 
Welcome back!
Is there exit(:normal) or exit({:shutdown, some_value}) somewhere in your code that could have been called by that process?
Or maybe it was linked to another process that exits with shutdown? I don’t remember how GenServer behaves in that case.
Most Liked
mudasobwa
There are two things to take into consideration as well:
① Restart strategies, specifically :max_restarts - the maximum number of restarts allowed in a time frame. Defaults to 3 and :max_seconds - the time frame in which :max_restarts applies. Defaults to 5. If something went south and there were 4 attempts to restart it within 5 seconds interval (defaults,) all failing, the GenServer would end up not restarted. This is likely your case, if e. g. the DB connection has dropped for 3 seconds. The culprit would be 500ms interval in init/1, which makes 4 consequtive insuccessful attempts to execute do_some_work/1 to fit into 5 secs and shutdown the process forever.
② The mailbox overflow. This technically might be your case, but extremely unlikely, just saying for the complete picture.
LostKobrakai
The app will terminate if the root pid exits, and immediatelly on the first exit, no restarts. And if the app is :permanent it’ll make the whole vm stop.
lud
Yes the supervisor will exit if it has to restart a child too many times. And it will be restarted by its own supervisor.
You can have a chain of Sup → Sup → Sup → Sup → Worker and all elements of the chain will behave like this.
A good way to think about this is that the code describing the supervisor tree (each supervisor children, and the children description in each child supervisor, and so on) describes “how the app state should be at runtime”. And the system will try its best to maintain that state, or otherwise fail and shutdown totally.
Of course that description can be altered at runtime by calling start_child, delete_child to add/remove parts of the tree, and of course having :transient, :temporary or :significant children.
But there is no way it will skip some parts of the tree unless you tell it to. Max-restarts and al’ will only have a temporary impact on the state of the system, but the app will always get back to the desired state.
So if my understanding is correct, if
MyWorkerwas linked to another process, and that process exited with a:shutdown(or{:shutdown, value}), then it would indeed causeMyWorkerto terminate normally as well. Unfortunately,MyWorkerisn’t linked to any other process.
If the child spec for MyWorker has restart: :transient then yes! (:shutdown is not “normally” but the supervisor will not restart it indeed).
Any progress on finding the cause?
- Another possibility is that the child spec is overriden somewhere and
:transientis replaced with:temporary; but I guess this is unlikely. - Are there any calls to
Supervisor.terminate_childorGenServer.stopin your codebase? - Are you using
:significantwith:any_significantor:all_significant?
Last Post!
smaximov
Unfortunately, no progress so far. Also, “no” to all of your points here.
For now we decided to change the restart type to :permanent and monitor this issue closely. I will follow up with more information if it arrives.
We didn’t find any suspicious entries in the logs (we know the approximate time the issue occurred from the application metrics). There were also no relevant errors reported in our Sentry instance in that time frame.
It’s possible, but very unlikely. There are maybe 2 people in our company other than me with the necessary knowledge required to gracefully terminate a supervised process, and the issue happened in the middle of the night. We also asked our Kubernetes admins to check if anyone accessed the pods in that time frame, just to be sure.
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"