Distributed application - network split

I have the same problem, I get all the nodes from the sys.config file and check if any lower node can connect to any higher priority nodes, if it can I use erlang:halt() to kill it. If I stop the app on the master server it failsover to the standby. If I start the app on the master it takes over and the standby stops, work fine if I use it as optional and failsover again if master stops. But I can’t get it to work so that when the master crashes the app failsover, heart gets in the way. It only works if master has no heart. Any advice?

That’s correct but often the heavy workers I have “pleasure” to work with are not confined within BEAM. Every time they do video or image processing, or spawning headless crawling web browsers for example - either external binaries or external C libraries are called and this actually has an impact on the rest of the system.

This is in fact the reason why the worker nodes would ideally be isolated.

One point to take into consideration here is: BEAM is really, really good at keeping everything responsive under stress. I mean, let’s say you start a heavy worker process that distributes the load in your cluster and starts competing for resources with your web app. The impact on response times for, say, REST calls wouldn’t be big. That’s because the BEAM is one of the single preemptive scheduling systems I’ve ever seen. Depending on the feature, you could just use this fact and just distribute the load in your cluster for all your process just the same.

I have no links to point you to but I’ve personally done some synthetic benchmarks comparing the BEAM and Node.js under stress. Suffice to say that BEAM under load (response times under heavy loads of requests) beats node pretty badly once the machines are on 90% percentile (even a little lower than that). BEAM had practically the same response times up until 99% of stress.

Swarm doesn’t support node specialisation, but that seems like a good feature to build. You could have configuration mapping a named cluster type (web, worker) to individual cluster config and then specify the type when starting up a registered a process.

Currently you would need separate clusters to restrict where processes can run.

perfect, thank you :).

If I go with Swarm, I’d have to make some architectural changes, however. Unit that’s being distributed across nodes is in this case process, versus OTP application.

I also can’t seem to find a way to make nodes specialized. For example, I want to run my web app on on some nodes and my heavy worker processes that crunch numbers on others. If I understand Swarm correctly, I’d run everything on all nodes, or create separate clusters for web interface and worker backend.

Someone’s written an EC2 cluster strategy for libcluster: GitHub - kyleaa/libcluster_ec2 · GitHub

Yes, I’m not looking at it from distributed state point of view at all just yet. As I originally pointed out, the objective is being able to fail over to backup nodes and recover from failure.

My current thinking is that the built-in dist_ac is not suitable for any cloud-based deployment, as it was designed for co-located servers. It does not handle well network splits, resulting in brief freezes of nodes when it happens, and then can’t re-join on it’s own the cluster and restore the topology of apps running on top of the cluster.

I am experimenting now with @bitwalker 's libcluster, as a low-level cluster manager. libcluster — libcluster v3.5.0 with possibly swarm running on top, or a custom solution. I’d need to implement EC2 clustering strategy, I think too.

You can use the Swarm library which provides a distributed process registry. I contributed changes to make it CP, rather than AP, so that it remains consistent during a network partition. This is achieved by using a static quorum distribution strategy as described by @CptnKirk. These changes have not yet been published to Hex and are available from the master branch in GitHub only. I plan to ask Paul to release v3.1.

During a network partition, processes are redistributed so that only one instance of each named registered process is running on the cluster. Those on the wrong side of the split are stopped. When the cluster reforms the processes are redistributed and you can handle migrating process state per GenServer (discard, hand-off, etc.).

Yep you are right! My first read led me to believe this was the first step that would eventually end up on the distributed state problem. But, that may not be the case \o/

All true, but none of that works if you can’t detect changes in topology. Especially for split-brain situations where you might double-owner yourself and start to corrupt that shared DB. It sounds like he’s having problems at that level. Getting the join/leave master election stuff working.