I run an agency and we have multiple projects. In most of them we actually don’t have to use any clustering, as it often happens we are running on a pretty standard single-database-multiple-web-nodes or even single-database-single-web-node set ups (yeh, Elixir is fast!). But there are projects where we do use clustering too.

Let’s start with use cases I see:

Actually the most common use case is that we want to have PubSub (as in Phoenix PubSub) functionality shared across the cluster, without the need for additional dependency (like Redis). This is used by default by Phoenix Channels, but not only. We also use Absinthe GraphQL server, and when you do real-time push updates this way (with GraphQL subscriptions), you often have to trigger these as well with a PubSub message, and you want all clients, connected to all nodes, to receive that update. The third, also PubSub-related use case us LiveView. It’s similar to GraphQL subscriptions in a way that whnever some event happens, as in certain record gets updated, all currently connected LiveViews should receive a messge and do something like render new version of updated record. We do that also with PubSub, and when our nodes are clustered this is a no-brainer in usage and configuration.

After PubSub, the second use case for clustering I think is the need to perform a cluster-wide lock, i.e. critical section. Or more generally speaking: limit the concurrency of something cluster-wide, to one or N concurrent actions of given kind. For example, if you track some usage of your system, and have pay-as-you-go or billing level plans, you want to warn user when they are approaching limit, and then maybe suspend or switch plan, or apply more charges once they exceeded usage. You may want, these events to happen precisely once. Or, you want to throttle the usage of the user when they are exceeding some limit of API calls. You usually can do these things without clustering, and rely on something like locking records in database but that has own disadvanteges like, well, that your database connections are locked for a long time for example.

Third thing are things that can go wrong across multiple nodes. For example, if you need a circuit breaker that’ll shut off some part of the system that contacts an API that started timing out, or a rate limit usage of external API across cluster. Again, it’s probably possible to do with something like Redis or database locking, but just so much easier and more natural to do it within Elixir processe in cluster.

And finally caching and keeping the system “hot”, as in warmed up after deployments. If you have the need to keep soemthing in memory (versus in database or external service) you can duplicate the thing on all of the nodes. You can do that without clustering. But you can also form a cluster and have cluster-wide cache, provided your node-to-node connections are fast (which they should be). Or, you can do a mixture of both things, keeping some things on hand, in memory, for all nodes in cluster, and other things local for particular node. Then, what becomes interesting is the ability to pass on that cache to newly started instances. If you release to prod often, you may find your system needing some “warmup” time, when it doesn’t yet know what’s going on, and has no caches, and is building it up as the clients make requests. This starting carte blanche style may not be desirable, and it can even lead to some serious performance issues if you deploy during the high traffic hours. So, by briefly forming a cluster between shutting down, and starting-up nodes, you can pass the relevant state, as in cache/counters/processes from old version of application to new version of application.

When it comes to what platform we use:

We have been doing that on dedicated hardware, EC2 instances (with ECS) and also recently on Gigalixir. The last option is definitely the easiest to set up as they have figured most of these things out, including the state passing form shutting down to stopping nodes is possible (which I learned only recently, silly me!). Unfortunately i have no experience with your stack :/.

Hope that’s helpful!

Well, I think we are mixing two concepts. There are at least two ways to do what we are talking about here.

1. Use Erlang / Elixir releases with hot code upgrade. I haven’t used this method in production to be fair. But it works when you have set of servers, where you deploy your application to, and the updated application will run on the same servers. By servers I mean OS instance, as in EC2 instance or a real hardware server, that stays the same during deployment. This method doesn’t really care about clustering, which may or may happen in parallel to hot upgrades. It’s just deploying new version of code, to the same servers that were running previous version of code, and there are hooks in GenServers and friends to handle state passing between ‘old version’ and ‘new version’ of code that was just deployed: mix release — Mix v1.20.2

Again, these days most of the things I work on are not deployed to such static/dedicated servers, but to a VMs created as needed by some piece of infrastructure, and discarded after the application shut downs. This is the way anything Docker-based or Kubernetes-based works.

2. Use clustering, no hot code upgrade but cluster starting and shutting down instances.

This is method suitable to pass the in-memory state on deployments when you use something like Kubernetes. When you deploy new version to the cloud, the old instance(s) of application is/are still running on their own containers. During deployment, the piece of infrastructure you use creates new containers for the new release. These start their own little OS instances and run application. Now, here’s the moment where your infrastructure may establish a link between legacy version of application and new version of application, so you can pass state. Again, Gigalixir does that by default, I believe.

We are always using GitHub - bitwalker/libcluster: Automatic cluster formation/healing for Elixir applications · GitHub here to handle cluster formation and this is not really important here.

What is important is that you can listen to events when new nodes join or leave cluster. And example code can be seen here: horde_connector/lib/horde_connector.ex at master · UrbanOS-Public/horde_connector · GitHub

So you monitor the nodes in your cluster in some process, and get events when new node joined/left and you can decide to pass a state to this newly started node by sending some process running on that node message with the state.

There are at least two projects that allow you to abstract most of the details here and do a lot of the legwork for you, one is Swarm another one is Horde. With both you can start processes on the nodes in the cluster, and they will react to cluster formation providing hooks to pass state. With Swarm it’s a bit easier (Swarm – swarm v3.4.0) but we observed some undeterministic behavior here, i.e. bugs. In theory it’s super sweet, however, and the API is really nice.

Then, you can do the same with Horde (GitHub - elixir-horde/horde: Horde is a distributed Supervisor and Registry backed by Postgres · GitHub) with a bit more of legwork State Handoff — Horde v0.10.0

It depends a bit on what you’re looking for with clustering. We are on AWS / EKS (aws managed kubernetes) and do use clustering. Our clusters are homogeneous, all nodes are running the same code / perform the same roles. The main value we get out of clustering is easy use of phoenix pubsub and other internode message passing.