vfsoraki

vfsoraki

I have been testing hot code reloading functionality of Erlang VM, using Elixir and making releases with exrm.

Here is the application module

defmodule RelTest do
  use Application

  # See http://elixir-lang.org/docs/stable/elixir/Application.html
  # for more information on OTP Applications
  def start(_type, _args) do
    import Supervisor.Spec, warn: false

    port = Application.get_env(:APP_NAME, :listen_port, 9000)
    {:ok, socket} = :gen_tcp.listen(port, [:binary, active: false, reuseaddr: true])

    # Define workers and child supervisors to be supervised
    children = [
      # Starts a worker by calling: RelTest.Worker.start_link(arg1, arg2, arg3)
      # worker(RelTest.Worker, [arg1, arg2, arg3]),
      worker(Task, [fn -> TestListener.start(socket) end])
    ]

    # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: RelTest.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

And here is the worker

defmodule TestListener do
  require Logger

  def start(socket) do
    {:ok, client} = :gen_tcp.accept(socket)
    Logger.info "A client connected"
    Task.async(fn -> loop(client) end)
    start(socket)
  end

  def loop(socket) do
    case :gen_tcp.recv(socket, 0) do
      {:ok, _} ->
    say_hello(socket)
    Logger.info "Said hello to client ;)"
    loop(socket)
      {:error, _} ->
    Logger.info "Oops, client had error :("
    :gen_tcp.close(socket)
    end
  end

  def say_hello(socket) do
    :ok = :gen_tcp.send(socket, <<"Hey there!\n">>)
  end

end

This is version 0.1.0. So I run these:

MIX_ENV=prod mix compile
Mix_ENV=prod mix release

and I get a nice release. I run it with ./rel/rel_test/bin/rel_test console and everything works. Now I’m going to bump the code and version, so here is the version 0.1.1 of listener:

defmodule TestListener do
  require Logger

  def start(socket) do
    {:ok, client} = :gen_tcp.accept(socket)
    Logger.info "A client connected"
    Task.async(fn -> loop(client) end)
    start(socket)
  end

  def loop(socket) do
    case :gen_tcp.recv(socket, 0) do
      {:ok, _} ->
    say_hello(socket)
    Logger.info "Said hello to client ;)"
    loop(socket)
      {:error, _} ->
    Logger.info "Oops, client had error :("
    :gen_tcp.close(socket)
    end
  end

  def say_hello(socket) do
    :ok = :gen_tcp.send(socket, <<"Hey there, next version!\n">>)
  end

end

Now I run

MIX_ENV=prod mix compile
Mix_ENV=prod mix release

and appup is created successfully, then to do hot upgrade

./rel/rel_test/bin/rel_test upgrade "0.1.1"

and the upgrade works, but it kills my listener after upgrade.

I tested with a nc localhost 9000 (9000 is the port of listener), staying connected and running upgrade command. Connection gets killed and I get a message in console:

=SUPERVISOR REPORT==== 31-Aug-2016::23:40:09 ===
     Supervisor: {local,'Elixir.RelTest.Supervisor'}
     Context:    child_terminated
     Reason:     killed
     Offender:   [{pid,<0.601.0>},
                  {id,'Elixir.Task'},
                  {mfargs,
                      {'Elixir.Task',start_link,
                          [#Fun<Elixir.RelTest.0.117418367>]}},
                  {restart_type,permanent},
                  {shutdown,5000},
                  {child_type,worker}]

So why this happens? Is it something I’m missing, or it is the expected behavior? Is it not the use case of hot code reloading?

I have read LYSE, but the author says the running code should keep running, only the external calls made after the upgrade are to be served with new version.

Maybe Elixir’s releases are not supposed to work like raw Erlang’s, but anyway in the end, both are running on BEAM, aren’t they?

Then why kill the worker?

P.S. Here is the SO question just in case.

Showing Posts 1 to 4

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I am by no means an expert on hot code reloading but it may be related to the fact that you aren’t using proper OTP children. A task is not an OTP behaviour and is not a candidate that is safe to hot reload. Your looping code isn’t either.

@fishcakez would know more but it strikes me that this is relevant.

fishcakez

fishcakez

Ecto Core Team

Likely your process is getting killed because the upgrade instructions define a brutal purge on post purge. That is kill all old process running old code once the new code has been loaded and then purge the old code. This means we enter a race condition where all process for that module must get on to the new code as soon as it is loaded - otherwise killed.

With a GenServer, or other OTP process, it is possible to use an update instruction that will suspend all the GenServers currently running the module and then resume them after loading. This should mean that none of the GenServers should be running the old code and do not get purged. Of course any new GenServers started during the upgrade may get killed.

However when not doing an update with an OTP callback module the post purge is likely to kill any processes running long running functions. It is possible to do a soft purge instead that will only purge the old code once processes stop using it. Instead of killing all process using it immediately and then purging. Also an update isn’t occur but just a load_module. The safest way to use this is to keep the external contract the same (keep the arguments and return the same types) and consider a soft_purge for the after purge if possible. This is exactly the type of loading you will do when using Phoenix/Plug on the non-OTP HTTP processes.

In the above example the functions do not have an opportunity to use a new version of the code so require soft_purge. However a second appup will kill the processes as a third version of the same module can not be loaded - the process calling :gen_tcp.accept will be using the initial version of the module. This can be avoided by calling __MODULE__.start(socket) and __MODULE__.loop(socket) as the latest version of the code will be loaded for external calls. Whereas start(socket) will use the current version and never a newer version. The contract is remaining the same as the argument is always a socket and return is eventually :ok/ignored.

After applying that simple change then appup would need an entry like:

{load_module, 'Elixir.TestListener', brutal_purge, soft_purge, []}

Note that it seems like you want Task.start_link and not Task.async as never await the result. This will cause the acceptor’s message box to grow as an :ok and :DOWN message will be sent for every connection and never received. Also it is common practise to have one module per process as different module loading schemes might be needed for different processes. For example the recv/send processes may become a GenServer and require update.

vfsoraki

vfsoraki OP

Pretty nice explanation. Thanks!

Just to be sure that I understood, here is the simple version:

exrm detects OTP behaviors when generating appups, and generates an appup that calls the code_change callback (or similar).
But when there is no OTP behaviors, it just does a brutal_purge and kills all old-code-running processes. To prevent this, one has to change appup himself/herself, change brutal_purge to soft_purge and makes sure that code is using external calls, i.e. __MODULE__.fun().

fishcakez

fishcakez

Ecto Core Team

Yes but to clarify, the soft_purge is for the post purge (there is also a pre purge) but you may require the brutal_kill to ensure no old code is running. Also it is the main loop function for a process where the external call might be needed. In most situations it is not a requirement.

— All posts loaded —

Where Next? Top

Trending in Questions Top

stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New
water
I’m aware of 2 such projects, are there others? git: GitHub - team-alembic/realworld: A fullstack Phoenix LiveView application with back...
New
itzmidinesh
Hi all, I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
peck
Hi everyone. My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
leoschweizer
Hi, we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume ...
New
csmr
Hi all, I’m new here and would like to hear peoples views on my Elixir/Phoenix based game-engine! :smiley: Mutonex gamedesign was finish...
New
Stef-Verniers
Hello I’ve been coding in Elixir for almost two years now because of my current job where we implemented Elixir in our application. But ...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
webofbits
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself. My main conc...
#ai
New
kip
In 2021 I started a new library called Tempo with the objective of modelling time as a set of intervals - not as instants. In 2022 I gave...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews