Distributed application - network split

Split brain scenarios like these are a problem for stateful distributed systems. The dirt simple solution is to detect a split by detecting that your current cluster only has < (total / 2) + 1 nodes in it. If so, you’re on the wrong side of the split. Shut yourself down, let something like heart start you back up. Then wait to rejoin the cluster using the normal cluster joining rules you’ve defined for your app.

A more complicated solution might involve only shutting down the request processing tree of processes responsible for handling reqs that could cause damage. Leaving a stateless tree open for better administration and triage.

The biggest problem about distribution is state. If your cluster needs a distributed state (which is not always the case) then you will probably need some distribution strategy.

For example, if you want the state of the cluster to be always available, then every time something gets written to that state, you would replicate it. Then you would probably meed a consensus algorithm (raft, paxos), a distributed hash ring and other tricks to have everything stable. This is the case for something like Lasp or Riak Core.

But, if you can save your state in a DB (even your cache, authentication and whatnot) then that state is not your application worries. So, that is why there is no ready available solution for all use cases.

All true, but none of that works if you can’t detect changes in topology. Especially for split-brain situations where you might double-owner yourself and start to corrupt that shared DB. It sounds like he’s having problems at that level. Getting the join/leave master election stuff working.

Yep you are right! My first read led me to believe this was the first step that would eventually end up on the distributed state problem. But, that may not be the case \o/

You can use the Swarm library which provides a distributed process registry. I contributed changes to make it CP, rather than AP, so that it remains consistent during a network partition. This is achieved by using a static quorum distribution strategy as described by @CptnKirk. These changes have not yet been published to Hex and are available from the master branch in GitHub only. I plan to ask Paul to release v3.1.

During a network partition, processes are redistributed so that only one instance of each named registered process is running on the cluster. Those on the wrong side of the split are stopped. When the cluster reforms the processes are redistributed and you can handle migrating process state per GenServer (discard, hand-off, etc.).

Yes, I’m not looking at it from distributed state point of view at all just yet. As I originally pointed out, the objective is being able to fail over to backup nodes and recover from failure.

My current thinking is that the built-in dist_ac is not suitable for any cloud-based deployment, as it was designed for co-located servers. It does not handle well network splits, resulting in brief freezes of nodes when it happens, and then can’t re-join on it’s own the cluster and restore the topology of apps running on top of the cluster.

I am experimenting now with @bitwalker 's libcluster, as a low-level cluster manager. libcluster — libcluster v3.5.0 with possibly swarm running on top, or a custom solution. I’d need to implement EC2 clustering strategy, I think too.

Someone’s written an EC2 cluster strategy for libcluster: GitHub - kyleaa/libcluster_ec2 · GitHub

perfect, thank you :).

If I go with Swarm, I’d have to make some architectural changes, however. Unit that’s being distributed across nodes is in this case process, versus OTP application.

I also can’t seem to find a way to make nodes specialized. For example, I want to run my web app on on some nodes and my heavy worker processes that crunch numbers on others. If I understand Swarm correctly, I’d run everything on all nodes, or create separate clusters for web interface and worker backend.

Swarm doesn’t support node specialisation, but that seems like a good feature to build. You could have configuration mapping a named cluster type (web, worker) to individual cluster config and then specify the type when starting up a registered a process.

Currently you would need separate clusters to restrict where processes can run.

One point to take into consideration here is: BEAM is really, really good at keeping everything responsive under stress. I mean, let’s say you start a heavy worker process that distributes the load in your cluster and starts competing for resources with your web app. The impact on response times for, say, REST calls wouldn’t be big. That’s because the BEAM is one of the single preemptive scheduling systems I’ve ever seen. Depending on the feature, you could just use this fact and just distribute the load in your cluster for all your process just the same.

I have no links to point you to but I’ve personally done some synthetic benchmarks comparing the BEAM and Node.js under stress. Suffice to say that BEAM under load (response times under heavy loads of requests) beats node pretty badly once the machines are on 90% percentile (even a little lower than that). BEAM had practically the same response times up until 99% of stress.