Available choices to keep state across deployments?

thbar · June 25, 2022, 9:38am

Hi!

My 14-yo son and I are playing around with a multiplayer online game idea. Phoenix & Elixir are a good fit for the task, so here we go.

The app could be deployed on fly.io or gigalixir, typically.

An important part of the game is that round can take a bit of time (e.g. 5 minutes), and we do not want players to see their game interrupted because we deploy a change.

Initially the game won’t have a “lobby” (place to gather before launching a game), but we will likely add that later.

I wonder what are the available options in 2022.

I have thought about these ones:

storing the state in Ecto/Postgres is perfectly valid for the type of game
hot upgrades could be doable in that case (the upgrade path is quite clear, the app quite simple), maybe coupled with some form of state snapshot (mnesia etc)

I’d be happy to get some insight from people already doing this today.

Thanks!

BartOtten · June 25, 2022, 10:13am

Another option is to separate the app with state from the one with logic, making them talk to each other using messages. Then only the logic app is restarted and no state is lost.

However, since Postgres is an option I would recommend that.

ps. Don’t forget that Mix releases do not support hot upgrades. Distillery does support hot upgrade releases.

thbar · June 25, 2022, 12:30pm

I had forgotten about that, so thanks for the reminder.

The relevant doc is here.

The idea of splitting logic and state is a good one, yet too much work at this point (but I’ll make sure to separate the code properly should we get there).

All in all I think I agree with you, just going DB given the scale is probably perfectly fine. Thanks!

sorentwo · June 25, 2022, 4:12pm

Fly does zero-downtime blue/green deployment by default. If retaining state isn’t absolutely critical you could attempt a handoff from the old app to the new app before shutdown.

Another option is to use DETS and write data to disk. Fly has persistent volumes for apps that you can provision.

However, I see no reason not to use Postgres for your son’s use case

msimonborg · June 26, 2022, 12:17am

I’m currently doing exactly this with a game I’m developing and deploying to fly.io. Here are the design considerations I’m working with:

Assign a UUID to each game and store it in the process state
For extra guarantees in a cluster you might try to globally register the process using the UUID so it can’t accidentally be started twice on different nodes (I’ve been using Horde.Registry but currently exploring alternatives)
Trap exits and save the state in the Postgres DB on any exit that isn’t :normal (or whatever other reason not to save, for example I use an :expire reason that is considered the end of the game so there’s no handoff)
Monitor the game from the client and keep a reference to the game_id so the client can restart the game if it goes down for certain reasons
Always check the DB in init/1 to see if a state handoff exists and load it when possible
Use bluegreen deployment strategy so you know there is zero downtime
EDIT: If you need your new VMs to connect to your old VMs during the deployment, then you’ll need to set a static RELEASE_COOKIE

It gets a little trickier if you are clustering the application and distributing it in geographically distant regions. There might be race conditions where the game could restart on another node before the data becomes available in the local read replica. I have a solution that’s working for that which I’d be happy to share if it’s relevant to you

Edit: Another option would be to just persist the game state on every state change so you don’t have to worry about trapping exits and the terminate callback. This might be especially useful if there are a limited number of changes per game. In any case, I have a GenServer doing periodic cleanup of stale game states in the DB.

thbar · June 27, 2022, 3:06pm

Thanks @BartOtten @sorentwo and @msimonborg for the input, all valid and interesting!

shahryarjb · June 27, 2022, 3:23pm

If you want to move a library instead of a module please see these lines:

github.com

mishka-group/mishka_installer/blob/25754feda20121f6f98d510c98a6ae98098f95c4/lib/installer/run_time_sourcing.ex#L103-L114


      
          def prepend_compiled_apps(files_list) do
            files_list
            |> Enum.map(
              &{String.to_atom(&1),
               Path.join(get_build_path() <> "/" <> &1, "ebin") |> Code.prepend_path()}
            )
            |> Enum.filter(fn {_app, status} -> status == {:error, :bad_directory} end)
            |> case do
              [] -> {:ok, :prepend_compiled_apps}
              list -> {:error, :prepend_compiled_apps, :bad_directory, list}
            end
          end

and

github.com

mishka-group/mishka_installer/blob/25754feda20121f6f98d510c98a6ae98098f95c4/lib/installer/run_time_sourcing.ex#L176-L179


      
          with {:load, :ok} <- {:load, Application.load(app)},
               {:sure_all_started, {:ok, _apps}} <-
                 {:sure_all_started, Application.ensure_all_started(app)} do
            {:ok, :application_ensure}

I bypassed this in my plugin, and it is working fine, but I did not write a document for it yet…

evadne · June 27, 2022, 5:45pm

Hi

If you are able to segregate state based on how much staleness can be tolerated:

Tier 1: No staleness allowed, data drives active game session, must be hot and accessible all the time to ensure proper gameplay
Tier 2: Staleness is ok, metadata around lobbies and games, not updated every second
Tier 3: Historical game data

Etc.

Then you could combine them and implement some kind of write-through cache with checkpoints.

Also worth considering, if your deployments are infrequent, and game lobbies do not last long, then simply drain each old instance of the app of all active connections, and eventually you will be able to retire these instances. This design allows you to not have to hand over so frequently.

e.g.

Deploy Version 1 to entire cluster of 2 Nodes
Lobbies started based on code in Version 1 on both Nodes
Deploy Version 2 to 2 new Nodes, set existing 2 Nodes to drain-only, no new lobbies can start
Wait until the games complete on the old Nodes then recycle them

vs.

Deploy Version 1 to entire cluster of 2 Nodes
Apply hot upgrade for Version 2

vs.

Deploy Version 1 to entire cluster of 2 Nodes
Replace each node, games have to migrate to valid hosts (this can take a minute or so)

I think it does boil down to whether the game clients can tolerate handover at all (think shooters vs puzzles) and also it is better to have a design where if you borked the release it does not crash everything.

BartOtten · July 4, 2022, 3:55pm

Just for reference: share the state in a cluster.

Make sure there is always one instance with the data