Distributed application - network split

hubertlepicki

So I have been investigating the feasibility of employing Erlang’s native distributed application architecture (Distributed Applications — Erlang System Documentation v29.0.2) for a project. I have done some research but I do lack hands on experience.

I am mostly interested in built in failover and takeover mechanisms, I think it will work just fine in my case. The problem are network splits.

I will be running the cluster in the cloud (EC2) and, I am afraid, the connectivity between those two nodes is not as guaranteed as it would be if the nodes were connected by a physical cable in the same date center. The connectivity will and does go down on occasion. The default dist_ac’s behavior is less then ideal:

nodes disconnect from each other
they keep running as if they were single nodes in the cluster.

To make things even worse, re-joining the cluster afterwards does not resolve the situation and failover / takeover no longer take effect.

I can detect that a node was disconnected from the cluster, but that’s about it I can do. I can’t find a way to recover from this situation - other than shutting down and starting the whole cluster from scratch.

Any recommendations you may have?

12 comments

#erlang #distributed-systems

4 4567 12

2018-04-16 23:53:31 UTC

First Post!

Show Best Posts
Show All (oldest first)
Show All (newest first)

CptnKirk

Split brain scenarios like these are a problem for stateful distributed systems. The dirt simple solution is to detect a split by detecting that your current cluster only has < (total / 2) + 1 nodes in it. If so, you’re on the wrong side of the split. Shut yourself down, let something like heart start you back up. Then wait to rejoin the cluster using the normal cluster joining rules you’ve defined for your app.

A more complicated solution might involve only shutting down the request processing tree of processes responsible for handling reqs that could cause damage. Leaving a stateless tree open for better administration and triage.

Post #1

Most Liked

slashdotdash

You can use the Swarm library which provides a distributed process registry. I contributed changes to make it CP, rather than AP, so that it remains consistent during a network partition. This is achieved by using a static quorum distribution strategy as described by @CptnKirk. These changes have not yet been published to Hex and are available from the master branch in GitHub only. I plan to ask Paul to release v3.1.

During a network partition, processes are redistributed so that only one instance of each named registered process is running on the cluster. Those on the wrong side of the split are stopped. When the cluster reforms the processes are redistributed and you can handle migrating process state per GenServer (discard, hand-off, etc.).

Post #5

slashdotdash

Someone’s written an EC2 cluster strategy for libcluster: GitHub - kyleaa/libcluster_ec2 · GitHub

Post #7

hubertlepicki

That’s correct but often the heavy workers I have “pleasure” to work with are not confined within BEAM. Every time they do video or image processing, or spawning headless crawling web browsers for example - either external binaries or external C libraries are called and this actually has an impact on the rest of the system.

This is in fact the reason why the worker nodes would ideally be isolated.

Post #11

Last Post!

romanr321

I have the same problem, I get all the nodes from the sys.config file and check if any lower node can connect to any higher priority nodes, if it can I use erlang:halt() to kill it. If I stop the app on the master server it failsover to the standby. If I start the app on the master it takes over and the standby stops, work fine if I use it as optional and failsover again if master stops. But I can’t get it to work so that when the master crashes the app failsover, heart gets in the way. It only works if master has no heart. Any advice?

Post #12