vrod

vrod

Saving and restoring process state after crash

I am trying to understand more about best practices with Elixir and processes. I thought about a GenServer that would maybe sometimes crash just like more complex real-world examples.

defmodule Dynamite.BadCounter do
  @moduledoc """
  A counter that sometimes crashes (pretend this is more complex process)
  """
  use GenServer

  @defaults [n: 0]

  def start_link(opts \\ []) when is_list(opts) do
    state =
      @defaults
      |> Keyword.merge(opts)
      |> Enum.into(%{})

    GenServer.start_link(__MODULE__, state)
  end

  @impl true
  def init(state) when is_map(state) do
    Process.send(self(), :increment, [])
    {:ok, state}
  end

  @impl true
  def handle_info(:increment, %{n: n} = state) do
    case Enum.random(1..10) do
      1 ->
        raise("Boom!")

      _ ->
        IO.puts(n)
        Process.send_after(self(), :increment, 300)
        {:noreply, %{state | n: n + 1}}
    end
  end
end

I can put this into application.ex as children, and then the process will restart, but always starts at 0 because the state becomes forgotten.

I have been reading about how to save the state in event of crash. For example GenServer with supervision tree and state recovery after crash - Bounga’s Home and https://www.codementor.io/@brunoripa/concurrent-programming-in-elixir-lyv2f6zhv

The common solution is to use another process to store the state.

However, this feels strange: the one process needs to know about the other process. It creates a dependency. The pattern looks similar to other languages where maybe a database or similar thing would be used to save the state.

I was more expecting to see this saving and restoring logic somewhere in the supervisor. My thinking is that my GenServer has only one job: to count. However the examples I have found so far want to make it have 2 jobs: to count AND to store and restore its state.

I hope I have explained this well. Does my confusion make sense? Is there some way to make the Supervisor handle the storage? Or am I just thinking about this wrong?

Thank you for you explanations!

Most Liked

John-Goff

John-Goff

Saving and restoring state after a crash is difficult, because often times it is the fault of incorrect state that your genserver crashed in the first place. If you automatically save invalid state and then load it again when your process restarts, you’ve just created an endless loop of crashes. So supervisors will restart their processes with “known good” state. Since the only state that we can guarantee with 100% certainty is “good” is the initial state (because if it wasn’t then the process would crash on init and you’d see it), that’s the default state to give to a newly restarted process.

al2o3cr

al2o3cr

IMO this is the trouble spot - a GenServer’s job is to maintain its state and handle messages related to that state. Guarantees about durability are part of that job.

The ecosystem offers a whole range of options for providing that durability:

  • a process is the simplest option, but only durable for the lifetime of the VM
  • ETS is a more-structured form of “use a process”, but still only in-memory.
  • DETS can fix that by storing data on local disk (with some gotchas)
  • separate systems (Redis, Kafka, Postgres, etc) can provide permanent storage, at the cost of additional complexity

A supervisor can have many children, so putting this logic in the supervisor would potentially be a bottleneck. Dealing with persistence in the children avoids this.

dimitarvp

dimitarvp

As @John-Goff said, there’s no guarantee that the last known state that your GenServer possesses is valid, hence OTP took a defensive stance and only ever restarts an actor with its initial state.

But if you are very convinced that you can guarantee last good valid state, you can do something like this in your GenServer:

def init(...) do
  state = load_persisted_state(...) # load last known good state from persistent DB
end

def handle_call(...) do
  state = ...
  do_stuff_with(state)
  # if the code hasn't crashed at this point
  # then the current state can be assumed safe.
  persist(state) # store last known good state to persistent DB
end

Personally I wouldn’t recommend it for a simple counter. But if you want it to be persistent, you can use DETS or Mnesia to load/persist it as shown above. Or Redis if you already have it in your stack.

Where Next?

Popular in Questions Top

chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42920 311
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30877 112
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New

We're in Beta

About us Mission Statement