Graceful Shutdown of GenServers

sheharyarn

I writing an Elixir app with GenServer that starts an external application on boot and shuts it down and does other clean-up on exit. I’ve added bootup functionality in the init/1 callback and cleanup code in the terminate/2 callback.

The init code works fine when the GenServer is started, and the terminate method is also called when the :stop signal is manually sent, but in the cases of unexpected shutdowns and interrupts (as in the case of hitting Ctrl+C) in IEx, the terminate code is not called.

What’s the proper way of doing cleanup when my Elixir app crashes or is unexpectedly shutdown?

Here’s the parallel StackOverflow Question and my code:

defmodule MyAwesomeApp do
  use GenServer

  def start do
    GenServer.start_link(__MODULE__, nil)
  end

  def init(state) do
    # Do Bootup stuff

    IO.puts "Starting: #{inspect(state)}"
    {:ok, state}
  end

  def terminate(reason, state) do
    # Do Shutdown Stuff

    IO.puts "Going Down: #{inspect(state)}"
    :normal
  end
end

MyAwesomeApp.start

11 comments

#genserver

0 8107 11

2017-04-06 21:48:27 UTC

Showing Posts 1 to 10

Show Best Posts
Show All Posts (oldest first)
Show All Posts (newest first)

OvermindDL1

The ‘proper’ way to shut down a node from the console (or in code) would be :init.stop(), which can take an optional integer that is the return code of the program. Ctrl+c is a harsh stop (I wish iex would bind it to :init.stop() with, say, a 30 second timeout that counts down on the screen, maybe displaying what it is waiting on to top too).

Post #1

sheharyarn

So there’s currently no way to catch Ctrl+C (and other sudden) exits?

Post #2

sasajuric

Author of Elixir In Action

To increase chances of the terminate callback being invoked, the server process should trap exits. However, even with that, the callback might not be invoked in some situations (e.g. when the process is brutally killed, or when it crashes itself). For more details see here.

As mentioned, if you want to politely shutdown your system, you should invoke :init.stop, which will recursively shutdown the supervision tree causing terminate callbacks to be invoked.

As you noticed, there is no way of catching abrupt BEAM OS process exits from within. It’s a self-defining property: the BEAM process terminates suddenly, so it can’t run any code (since it terminated) Hence, if BEAM is brutally terminated, the callback will not be invoked.

If you unconditionally want to do something when BEAM dies, you need to detect this from another OS process. I’m not sure what’s your exact use case, but assuming you have some strong needs for this, then running another BEAM node, on the same (or another) machine, could work here. Then you could have one process on one node monitoring another process on another node, so you can react even if BEAM is brutally killed.

However, your life will be simpler if you don’t need to unconditionally run some cleanup logic, so consider whether the code in terminate is a must, or rather a nice-to-have?

Post #3

sheharyarn

That’s a pretty detailed answer that covers most of my concerns. Would you consider also posting this as an answer on my StackOverflow Question - for future reference?

Post #4

sasajuric

Author of Elixir In Action

Sure, I copy-pasted it there.

Post #5

pdawczak

That’s absolutely great answer - thanks @sasajuric. It confirms this unfortunate situation - what approach would you suggest then for handling ports closure upon killing iex?

I’m opening port to boot executable up when elixir application is started (https://ngrok.com/ to be more precise). This is used for development purposes, so surely, the way you use this dev-env is, if you need to reboot the app - you’re killing it (Ctrl-C) and iex -S mix again, but this crashes as the executable has not been killed along the way.

The way I was trying to handle that, was to send kill -9 to process from terminate, but this is not invoked…

Thanks for your advice!

Post #6

pma

If the Erlang VM stops, the external port process will receive a signal indicating that the stdin closed.

If ngrok doesn’t gracefully stop on CTRL+D, you can use a small bash wrapper to handle the stdin closing and then killing ngrok with the appropriate signal.

Post #7

pdawczak

Thanks @pma!

Yes, indeed - ngrok (when watching: watch -n 1 "ps ax | grep ngrok") seems to disappear after some delay after killing iex. Unfortunately, it is not instantaneous and it’s indeterministic - it’s alive from couple sec to up to 30 sec, and just then disappears.

This is a problem, because if you’re trying to kill (eg. Phoenix dev server) and start again shortly after - everything crashes, as new ngrok can’t be opened (unless the previous one dies quickly enough before attempting to open a fresh one).

I also tried a little wrapper script (like the one described here). This, in fact, kills ngrok along exiting iex, but it suffers another problem - when ngrok crashes (for whatever reason, I’ve tried to mimic that by killing - kill -9 the ngrok process), this doesn’t propagate up via port, as the pid of running process points to the wrapping script.

Post #8

sasajuric

Author of Elixir In Action

If the Erlang VM stops, the external port process will receive a signal indicating that the stdin closed.

This is the correct hint. The external process will get EOF on its stdin. However, if the external program is busy doing something else, it could linger for much longer before it detects that. I’ve written a bit about this here (see “Program Termination” section).

IMO, the cleanest solution, if you own the code of the external program, is to adapt it to run processing in a separate thread, while the main thread just does I/O. That way, the external program can immediately detect the termination of the other side, and terminate itself immediately.

If that’s not an option, I think (but not sure) that Porcelain by @alco might offer some automagical help.

Post #9

pdawczak

Thank you @sasajuric, this is very valuable answer!

I’ve read through your article earlier, when I was starting playing with ports and it brought a lot covering basics. Great post!

You’ve stated:

It’s worth noting again, that a port is closed when the owner process terminates

This is the bit I wasn’t sure - because iex session is killed (this is what happens with Ctrl-C + Ctrl-C, right?) does Elixir have time to send EOF?

Unfortunately, ngrok is application I don’t own and I had suspicion what happens, but you’ve brought final confirmation. As mentioned earlier, wrapping with script was causing problem of not propagating it stopped and as such, it was difficult (impossible?) to detect it and supervise - that was the reasoning I’ve decided to go without wrapping script.

Thank you for your input! I truly appreciate it!

Post #10

Load more posts (1 remaining)