Hi! Just a few thoughts on this.

Re: seems a bit too fragile
Process state is fragile, period, even without the distribution. It’s just a property of OTP that processes (along with their state) may go away because we “let it crash”. Therefore, it’s important to design our applications to be resilient to that. So no, there’s no “guarantee” that the technique I described can preserve state, and I don’t know that there is (or even should be) a way around it. If you have critical state and need guarantees, use a database.

Re: other patterns
The CRDT method of preserving state that I described in the talk, is just one option. It works well for simple cases and is wicked-fast (and is easy to deploy, and makes a great demo…,) but it may have trouble scaling to large clusters. As you mentioned, you could also stash it in an external key-value store like Redis, or maybe even a queueing or pubsub system. (I’m actually currently investigating the latter myself.)
Overall, we are still pretty early in the development of best practices around this problem. The point I was trying to make in my talk was that it’s a hard problem, but there are solutions and ideas out there to explore, and it’s important for us in the community to be trying things out and sharing our findings.

Re: public code
The code I used for the demo in my talk is at GitHub - ElixirSeattle/tanx: Tank game created by the Programming Elixir study group · GitHub but I haven’t updated it since ElixirConf and it’s a bit out of date. (Horde in particular has evolved since then.) It’s also not really well documented, sorry. However, Chirag Singh Toor did a detailed writeup of how he produced a similar setup. You might try looking through that.

Re: Gigalixir and containers best-practice
My understanding is that Gigalixir effectively mounts your app (release) as a volume in a container. So the container is used more like a VM, for the OS image and process isolation but not for the application image. So yes you can do hot code swapping there. Additionally, elixir releases can include ERTS, OTP, and the elixir runtime in the release itself. So you shouldn’t have to kill the container for a new elixir version. Only an OS update should require new containers. (@jesse may have more to say, or may need to correct some of my understanding here.)

That said, yes, techniques like this kinda-sorta go against current “container best practices”. However, I’d suggest not taking that too rigidly. We are, again, still very early in the development of the container ecosystem, and “best practices” are very much in flux. For example, the whole issue we’re dealing with here is that current container “practices” are tied to a “traditional” stateless web application model, whereas OTP explicitly breaks out of that mold. That’s our opportunity as Elixir and OTP developers: we have a disruptive technology that enables us to do new things, and we have the opportunity to develop new patterns to make containers work for us, even if that ends up looking different from what you see with traditional web apps.

Can you point me to some analysis on the performance issues of containerizing BEAM? (Benchmarks or reasons for performance reduction?) I’ve occasionally heard this claim and I’d like to understand where it’s coming from. Thanks!

I am not using anything special other than a Deployment resource on Kubernetes to run my applications.
But there are a couple of libraries worth mentioning making integrating Kubernetes into your application a walk in the park:

• GitHub - coryodaniel/bonny: The Elixir based Kubernetes Development Framework · GitHub is a library to support writing actual operators for more complex Kubernetes integrations
• GitHub - coryodaniel/k8s: Kubernetes API Client for Elixir · GitHub is an excellent Kubernetes API client
• GitHub - bitwalker/libcluster: Automatic cluster formation/healing for Elixir applications · GitHub supports Erlang cluster formation for Elixir applications, it also ships with a designated Cluster.Strategy.Kubernetes

And worth mentioning is supporting software that are not Erlang/Elixir specific, but really leverage your Kubernetes experience:

• GitHub - traefik/traefik: The Cloud Native Application Proxy · GitHub acts as a Ingress controller, http/s load balancer
• GitHub - cert-manager/cert-manager: Automatically provision and manage TLS certificates in Kubernetes · GitHub lets you automatically provision certificates (e.g. Lets Encrypt) to use with traefik
• GitHub - kubernetes-sigs/external-dns: Configure external DNS servers dynamically from Kubernetes resources · GitHub lets you provision external DNS providers for dynamic DNS management (think of declarative external DNS for your whole infrastructure out of Kubernetes/API)