There’s several implementations of leader election, but none of them is very clean or easy to use really. At the end of the day Erlang was created for systems where net splits are very unlikely, and I think most users still use it under this assumption, or defer to an external system (typically PostgreSQL) to ensure consistency.

The “traditional” option has been gen_leader. It’s been around for many years, used in Riak, model checked, etc. so it should be good at this point, but you have to be careful to use the right branch

https://twitter.com/potsdamnhacker/status/928483849394196480

https://github.com/abecciu/gen_leader_revival

There’s also a Locks library, which I believe was a refinement of the above. Not sure about production usage.

https://github.com/uwiger/locks

Riak worked on Ensemble too, but folded before it could become a solid library.

https://github.com/basho/riak_ensemble

More recently the RabbbitMQ folks announced a project to implement Raft. This is still WIP. It could be a great building block though, especially if it gets used in RabbitMQ which would give it much needed battle testing.

https://github.com/rabbitmq/ra

Putting aside leader election for a bit - Swarm recently got a quorum strategy which can help ensure a unique instance of a given registered process exists in the cluster. A big limitation though is you can’t resize the cluster easily.

https://github.com/bitwalker/swarm

Not sure how dynamodb does it, but riak_core works like:

• You set up an abstract space, these are basically ‘partitions’, more tends to be better until they get to hold too little. Your maximum number of nodes that can be joined is the partition count, so setting it to like 128 or so is usually overkill. The partitions form an indexable cyclic ring.
• There are ‘virtual nodes’, each holds a single partition. These virtual nodes are separated and distributed evenly around the number of actual nodes in your system (distributed to keep ‘nearby’ partitions in the ring to be on as many nodes as possible).
• The virtual nodes themselves are what do ‘work’. They receive and (optionally) respond to commands (not necessarily to the sender). Riak_Core I think comes with a default set of virtual nodes that act as a work queue so you can submit work to it like poolboy (the ‘command’ is just a function to execute in this case).
• Riak uses it to store key/value information in it, it uses a distributed hasher to uniquely define where the key belongs in the partition ring (this is all handled internally), it then acquires the virtual node that matches that location and sends the command to store it it in that one as well as in the adjacent few before and after (depending on the quorum configuration you have it set to) and those virtual nodes will do whatever you wanted with the key/value, such as store it, retrieve it, mutate it, etc…
• You can however do whatever you wish in a virtual node, from being a processing array to store/fetch data to whatever.
• Riak_core has a riak_core_bucket thing built in that acts as a simple namespace with configurable properties, good for storage of information about the ring itself or small amounts of other data (say configuration data).
• Actual nodes that host virtual nodes can be taken up and down as wished, the virtual nodes automatically migrate to actual nodes that have the capabilities they need.

The example of use is of course Riak itself, say you set a quorum of 3/2, so 3 nodes have to verify a data write before a write is successful (the write message returns) and 2 nodes have to verify a data read (and they must match, otherwise all 3 will be checked and the 3rd will be temporarily taken offline as it performs a full resync, this will not affect the uptime of anything and is usually pretty close to instant anyway). As storage happens the information replicates around the ring until they are all in synch (but certain virtual nodes basically ‘own’ certain keys). This means that processing is distributed around the node while knowing that the value you ask for is written correctly as well as is read up-to-date since they get serialized through the quorum processes.

There are a lot of other aspects of it and this is a high level overview, but it is quite nice. I’ve used Riak in the past for quite a number of things (though nowadays, to be honest, I’d just use PostgreSQL, it even has notification on change functionality and all).

EDIT: The intro post of Riak Core:

Well tiny Erlang community already maintains pretty much the most advanced commercially viable VM for distributed systems. CoreOS (or whatever the current name is) was a tiny company until few days ago when it was acquired by Red Hat etcd obviously benefited greatly from being part of k8s efforts though. This def. would take concentrated effort from larger players that have use cases for something like this e.g the likes of Toyota, bet365, Erlang Solutions, Ericsson, Pinterest etc. or alternatively by a startup that for example was building event processing PaaS.