Best way to persist a processes state?

IMO you should just store this state in either KV store or the DB itself. And you already have a mechanism in place to detect server shutdown so this should complement it nicely.

Too much work? Not sure I get why.

You don´t need to have it normalized… you can use whatever you are already using to identify the rooms/games, make a json field(jsonb type if postgres, jsonb blob if sqlite) that stores the whole nested state as it is.
you can cast whatever comes from the db to a struct with struct/2 or use a schemaless changeset, or an embed schema if you have any sort of nesting to help cast the nested stuff together.

Consider what I’d call “option 3B”: store the state in a database table that has a BLOB column with an :erlang.term_to_binary in it.

This avoids the extra complexity of serializing/deserializing into other shapes.

It does make the contents of the state very opaque to DB queries, since doing binary-to-term in SQL would be quite challenging.

Ye the information for one game would be spread all over the place - Which feels very inconvenient for later.

Somehow I was completely neglecting this option It is probably by far the simplest option right now - I believe I will never have to query something from the actual content of the blob, just the whole blob.

Consider this option as the best one for your use-case.

The idea with using small files to store the data is exactly the same as this one, but with the addition that eventually you will have to implement features of a database system yourself.

At least for me that’s the worse scenario possible.

1. there is no way to query the data in the future if needed.
2. there is no way to bulk update to reshape stale data.
3. if you’re using structs, you’re prone to errors with malformed structs(converting the binary back to term won’t call the macro that enforces the shape of a struct).

Valid criticisms but IMO that’s mostly a future concern. For the moment it seems that OP wants to make sure they have persistent actors (more or less).

I too would reach for actual predictable serializable format after this goal has been achieved btw.

Ye, you are right.

I was surfing on a wave of motivation - which was really enjoyable and I wanted to keep going So I primarily wanted a “quick” solution that satisfies the needs for now.

Honestly I am not so worried about querying:

1. The process state dump is very short-lived and is deleted from the database as soon as the process restarts. It is only fetched as a whole.
2. Historical Game data still has all the data, just not super accessible - if needed, I can run a migration to add one or two columns and have a small ETL that extracts the required data into the added columns. A “match history” is stored separately (game_id, player_id, date_played, team, won?), which already covers a lot of ground.

prone to errors with malformed structs

Ye, my main concern is versioning.

Adding my thoughts on this to help OP and future readers out. I have done the :erlang.term_to_binary cast into db column approach for storing game state before and it works lovely. You can even :zlib.zip/:zlib.unzip too, if you like. To the three things you’ve pointed out:

1. For my case at least, this isn’t an issue. YMMV, but I think it’s likely that, like OP, many people will be in an “I just need to get this working” state, so they won’t care about this. If you do ever need it, you can extract the fields you need as separate columns*. Then you can use a migration, written in Elixir, to migrate existing data. You even have options here too - you can do a “bulk” update where you change all the records at first app boot up or you can do a “live update” where you do this migration when records are accessed/written.

* Another possibility is reporting on the metric you’re interested in elsewhere. For example, when making a clone of the game Boggle, I stored boards and user’s submitted words as erlang binaries in blob columns. I wanted to do reporting on the number of “globally new” words found by users every time they finished a game. Rather than changing the structure of existing, working tables, I just did the reporting to Prometheus. This is a highly specific to me example, but I wanted to share other ways I think are valid to work around this issue.

2. It will certainly not be as easy as UPDATE SET field WHERE clause, but it can be done. How one does this depends on a lot (size of records/tables/concurrent users). When building a small game, chances are there are few users and hence large chunks of time you can afford to have an inconsistent DB. This is highly dependent on all of the factors mentioned above, because if your records are small, you have few users and few games, such a migration script and its associated inconsistency might last a few seconds on the extremely long side. Even if this isn’t the case, there are ways around that - they aren’t easy, but it is possible.

3. I’d like to hear more about this if you can give an example. I’m fairly new to Elixir and my understanding is that structs are basically just maps with syntactic sugar.

you need to keep in mind that it gonna be a migration that needs to, retrieve data from the db, transform it in elixir, write back to the db. it’s preferable for your migration to do all the work as just sql statements, sometimes you can’t avoid the back and forth but in this particular case you could avoid it by just making it a unstructured column that the db knowrs how to handle.

so, when you do %MyStruct{field_a: 1, field_b: 2} you’re actually calling a function MyStruct.__struct__(field_a: 1, field_b: 2), this function is defined when you do defstruct .... This function is the thing that ensures that the output map has all the fields of the struct and only those fields and no other stuff.
so if the struct definition changes between when you do term_to_ binary/1 and binary_to_term/1 you gonna end up with a map without the guarantees of the __struct__/1 function call. you can test this on iex:

defmodule A, do: defstruct [:a, :b]
bin = :erlang.term_to_binary(%A{})
defmodule A, do: defstruct [:a, :c]
:erlang.binary_to_term(bin)

the risk of doing that is that all function that you pattern match on %MyStruct{} gonna pass, but the struct can end up not having the shape that you expect.

if there is a need to use binary_to_term/1 with structs, I strongly suggest using Map.from_struct/1 and struct/2 functions to ensure at least a few of the guarantees of what you expect from a struct are kept.