What to use to rehydrate process state

I’d lean to Redis. Yes, it incurs some little extra overhead but my thinking would go like this: since you are already using Kubernetes, spinning one more container (that could actually be useful for the state of other services in the future) isn’t such a big deal and Redis itself is quite standard tool for these scenarios.

I’d usually lean all the way to Erlang/Elixir software when I am coding a deployment monolith (although in your case I’d likely go for something even simpler like CubDB) but when k8s is involved you might as well go for standard industry practices.

That’s interesting. From a quick glance CubDB seems to have no interest in offering a distribution solution, is that right?

This one might be interesting to you as well

https://github.com/erleans/erleans

No, it doesn’t. It’s more like a per-node pure Elixir mini DB.

That does look interesting but I feel like it’s a bit more frameworky than what I’m looking for. Thanks for the suggestion nonetheless!

I did some more research and came across DeltaCRDT which is what Horde is using under the hood.

Any experience with/opinion on that, @LostKobrakai/@dimitarvp?

I am afraid I can’t give you a very informed take. I used DeltaCRDT once in a hobby project and I liked it a lot; that being said, I’ve heard people being displeased with CRDTs in general and that you still have to resolve some conflicts manually (imagine if two clients without internet made slightly conflicting changes and then both tried to synchronize them after they got back online).

However, I was left with the impression (after I tinkered on my hobby project for a while) that the CRDTs are very applicable in many areas. It just seems that people want to try them in scenarios that are either impossible (parallel conflicting changes) or not meant for them (offline-first apps).

But again, take this with a big spoon of salt. I am not an expert, just a guy who decided to see what’s this thing so many people talked about. They don’t seem very applicable to decentralized / federated scenarios is all. I liked using CRDTs and I can see applications but can’t talk informed about subtler nuances.

There is the library unsplit to make resolving split-brain scenarios easier.
It was never released on Hex.PM and it is quite old but I do believe it is used in a couple of serious applications/libraries. As one example that comes to mind, the user-authentication library Pow uses it.

That’s what I was implicitly referring too when I wrote that “it’s not a dealbreaker”. There’s also reunion.

I haven’t looked at any of those in detail though, as I first wanted to check other options which seemed more promising at the time.

There might also be another approach to this that does not involve distributed state but rather leans on the more exotic features of the BEAM: hot code upgrades.

Of course they’re more finicky to get right than regular deployments, but if there are only a few processes that you need to handle state updates for it might be doable- provided that you don’t care about accessing the state from all nodes but only care about keeping the state alive on the node that is about to be updated.

Taking this train of thought a bit further while also playing to kubernetes’ strengths (because allegedly hot code upgrades are not a scenario that ephemeral containers are made for), you could consider deploying a sidecar container inside the app’s pod that could be responsible for polling for newer versions and writing them on a shared volume and then trigger the hot code upgrade from that second container again.

Phew… just writing all these makes one realize how complicated the whole process can get …but if you do need persistent state across deployments and would rather tackle hot code upgrades than distribution, then perhaps it’s another option to consider as well.

Some notes from a few prod services running elixir on k8s

If you need some data after restart, those data won’t be ephemeral anymore, and you have to deal with it as like other persistent data, but probably with different performance/reliability requirement in this case. You mentioned you need to handle split-brain scenarios - then how are you handling the split-brain scenario with other persistent data?

The easiest way is to use the same persistent data (e.g. database for example). This is good unless the “less” important data causes bottleneck. In that case, you may just run a separate instance of the same kind of storage - instead of introducing new kind of storage.

You mentioned Redis - that’s a good choice (super easy to run). However introducing new type of external system needs good justification - for both app developer and infra management.

One of my projects actually stores all data in the database. We had to make sure new cluster should not start the job before existing cluster stopped and pushed all data into the database. With some work, if you carefully divide features, you can actually achieve zero-down time with some degraded performance on some features requiring pass-over the data.

In another project - I simply use oban background job processing instead of running a long-running process and worrying about redeployment. This project has relatively low volume so the choice was obvious, and it works great.

Oh, and yes, we also hit the network split in the k8s which led to split erlang cluster I highly recommend to just stick with simple centralized storage, instead of baking some tools inside your app erlang cluster. You should avoid adding more responsibility and requirement to your application when it’s not the core job.

If you move to erlang cluster wth hot-code reloading, some problems will become very easy to solve - but you also need to solve new problems, which are already covered by k8s and your existing k8s setup (such as CI/CD). If your app’s scale and challenges can justify the investment, that would be great. In my case this was not an option.