Distillery hot-update for long running processes not working

I’ve been testing how hot updates behave for long running GenServers, but so far, the GenServer process is being killed whenever I upgrade. I have tried setting a higher :shutdown and adding Process.flag(:trap_exit, true) to the init callback function, but still without success.

Some relevant parts of my GenServer:

use GenServer, shutdown: 120 * 1000

  @doc false
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)

  def run_testapp do
    GenServer.call(__MODULE__, {:run_testapp}, 60 * 1000)

  @doc false
  @impl true
  def init(:ok) do
    Process.flag(:trap_exit, true)

  @doc false
  @impl true
  def handle_call({:run_testapp}, _from, state) do
    Process.sleep(24 * 1000)
    {:reply, %{a: 1}, state}

Upgrades work fine if the process is not running, but otherwise the running process is killed and I get

** (exit) exited in: GenServer.call(Testapp.Server, {:run_testapp}, 60000)
    ** (EXIT) killed
    (elixir 1.10.0) lib/gen_server.ex:1023: GenServer.call/3
iex(app@> Testapp.Server.run_testapp

Any ideas or resources that could help me?

At a low level module replacement only happens when you refer to the module explicitly. For example, calling MyModule.do_the_thing vs. calling do_the_thing from within MyModule. The first will cause a lookup in the code server for the latest version of MyModule. The latter will not.

Further, BEAM only allows 2 versions of one module to co-exist. If one module update happens, but some processes never call the fully qualified name before yet another version is deployed, then those processes are killed. That’s what sounds like is happening to you.

1 Like