larshei

Best way to persist a processes state?

Hey everyone!

Preface

Over the last days I started to create an online version of Doppelkopf, a german classical card game, using Elixir/LiveView. The project is still in a pretty basic shape and deployed on fly.io - feel free to check it out at fehlsolo.de

Players join rooms and then create games. Both rooms and games are just GenServers.

When a GenServer dies, it dumps its state to an Agent and tries to fetch that state when it restarts. Works quite well so far.

When a game ends, the final state is dumped to a file as a binary (:erlang.term_to_binary). The score screen then grabs the file and calculates scores etc - in future this allows to watch a replay of the game, take over control at any point, analyse the games and maybe even train some models with it … if I ever get that far

Challenge

When e.g. a new version of the app is deployed, all process state is lost and all rooms and games disappear. As the game results are also just stored in files in the container, they are lost as well.

Discussion

What would be your preferred way of persisting state between app restarts and for the “long run” (game history)?

Possible approaches?

What I thought about is:

Store the process state as files, but in a cloud storage
Not sure about the (dis)advantages of this approach.
Store the files to a fly volume
The fly docs themselves suggest this is not a good idea for anything that should be around “longer” - might be fine to persist process state during an update.
Store the process state in postgres as bson
What is the best way to turn the retrieved (nested) map from the database back into the structs of my application? I have Ash available, but so far, all the “models” are just structs, not Ash Ressources or Ecto.Schemas.
Create normalized models and store everything in the db
I really dont want to do this

How would you tackle this?

18 comments

#discussion #database #state #flyio

1 873 18

2025-01-03 01:37:28 UTC

Marked As Solved

cevado

You don´t need to have it normalized… you can use whatever you are already using to identify the rooms/games, make a json field(jsonb type if postgres, jsonb blob if sqlite) that stores the whole nested state as it is.
you can cast whatever comes from the db to a struct with struct/2 or use a schemaless changeset, or an embed schema if you have any sort of nesting to help cast the nested stuff together.

Post #3

Also Liked

al2o3cr

Consider what I’d call “option 3B”: store the state in a database table that has a BLOB column with an :erlang.term_to_binary in it.

This avoids the extra complexity of serializing/deserializing into other shapes.

It does make the contents of the state very opaque to DB queries, since doing binary-to-term in SQL would be quite challenging.

Post #4

cevado

you need to keep in mind that it gonna be a migration that needs to, retrieve data from the db, transform it in elixir, write back to the db. it’s preferable for your migration to do all the work as just sql statements, sometimes you can’t avoid the back and forth but in this particular case you could avoid it by just making it a unstructured column that the db knowrs how to handle.

so, when you do %MyStruct{field_a: 1, field_b: 2} you’re actually calling a function MyStruct.__struct__(field_a: 1, field_b: 2), this function is defined when you do defstruct .... This function is the thing that ensures that the output map has all the fields of the struct and only those fields and no other stuff.
so if the struct definition changes between when you do term_to_ binary/1 and binary_to_term/1 you gonna end up with a map without the guarantees of the __struct__/1 function call. you can test this on iex:

defmodule A, do: defstruct [:a, :b]
bin = :erlang.term_to_binary(%A{})
defmodule A, do: defstruct [:a, :c]
:erlang.binary_to_term(bin)

the risk of doing that is that all function that you pattern match on %MyStruct{} gonna pass, but the struct can end up not having the shape that you expect.

if there is a need to use binary_to_term/1 with structs, I strongly suggest using Map.from_struct/1 and struct/2 functions to ensure at least a few of the guarantees of what you expect from a struct are kept.

Post #11

cevado

it’s off-topic but I see migrations as db “book keeper”, any relevant change or important operation I like to keep it as migration for historical purposes. in case is something to be executed manually, i’d just add the timestamp of the migration to the schema_migrations table and that’s it. but i don’t think this applies here

Post #17

Last Post!

sleipnir

Hi there! Your project sounds like a lot of fun and a great use of Elixir/LiveView!

If you’re open to exploring alternative approaches, you might want to check out the Spawn Actor Framework. It’s designed for managing distributed state with actors, offering built-in state persistence and recovery mechanisms. It could simplify handling game states, even across app restarts or deployments.

Let me know if you’d like more details—happy to share!