That is actually an interesting question, how to handle state.
If you really need session-data to persist during reboot, it is not ephemeral anymore. That is neither good nor bad, but you need to worry about state
Running multiple instances accessing given state adds distribution to the list of your problems
I would take a look at an event-based state-handling. Since it is based on messaging, it fits rather perfectly into erlang. Upside is, every node keeps its own database, would suggest mnesia in that case.
Merging state seems easier to me than handling concurrent access to coupled state
I’m no Elixir devops guru, but I always thought that the classical BEAM answer to this problem is to not reboot at all, but to do hot code reloading.
Has that gone so far out of fashion that it’s considered a bad idea? If so, why? Something about hot code reloading itself, or is it just the assumptions of most deployment/orchestration software that “new version = restart”?
I think the common answer is relatively straightforward. It consists of two parts:
Making sure an upgrade is ‘hot-code backwards compatible’ with the previous version can get very tricky/hairy. I remember to have read somewhere that a few companies that rely on hot upgrades were spending ~50% of their time on making sure the upgrades were correct (only being able to allot the other 50% to building new features). And of course some things cannot be hot-upgraded ever, no matter how hard you try.
Some examples: Erlang/OTP upgrades, Elixir upgrades, libraries that moved modules around in a way that resulted in splitting/merging of OTP applications, and of course kernel/OS (security) upgrades and the likes.
If there is data which should never be lost, you should not rely only on hot upgrades because power failures or other problems that make the BEAM or the whole PC (have to) restart do sometimes happen.
So hot upgrades are great to ensure that during most upgrades, sessions (like an ongoing phone call or websocket connection) do not have to be broken.
Combining this with using a cluster with multiple machines where you disconnect single machines one by one (once they have no sessions left) to perform a ‘cold’ upgrade on that machine and only then reconnect, allows you to have very high and consistent uptime for your cluster as a whole.
But it is not a tool that is useful if if a cluster setup like described above would be overkill, too expensive or for another reason not a good fit for your project,
and does not obviate the need to persist the important parts of your data.
Yes, Erleans is more of a framework but because the issue requires a framework of a solution. Without the framework you not only have to handle the state storage but the process lifecycle within the cluster as well.
That doesn’t mean Erleans will fit your use-case, just saying its a problem that benefits from a framework, essentially a layer on top of the OTP framework.
A while back (~2 years ago) I was considering an alternate approach that did work as a library rather than a framework.
I never fully finished the work on it, as my priorities had to shift unfortunately, but it is possible to write a custom process registry that will do the dehydration/rehydration of processes for you: PersistentGenServer on GitHub and its ElixirForum topic.
Do note that by itself it only tackles the hydration part and not the ‘distributed persistence’ part. But maybe it can provide as useful idea or building block for someone’s project.
