Erlang/Elixir native Etcd, Zookeeper alternative

Can you elaborate on what Etcd and Zookeeper are and what about them you’d want in Erlang/Elixir?

A bit of copy pasta from Etcd:
etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader.

Your applications can read and write data into etcd. A simple use-case is to store database connection details or feature flags in etcd as key value pairs. These values can be watched, allowing your app to reconfigure itself when they change.

Advanced uses take advantage of the consistency guarantees to implement database leader elections or do distributed locking across a cluster of workers.

We could have Kafka like log store that’s native to the ecosystem so you could build event processing pipelines without external dependencies and with nice integrations with GenStage or just as simple distributed consistent fault tolerant Registry etc.

How does it differ from Riak (and it’s various parts) and which would you chose for which purposes?

I can’t say I am super familiar with Riak various parts I don’t think Riak KV has ability no notify on key changes and in general Riak is more of a Data Store vs etcd/Zookeeper are mostly used as HA distributed consistent config store. So Riak is more of an end product vs etcd/Zookeeper is an infrastructure piece. Plus it used to be that Riak needed a custom build of erlang VM not sure where that stands.

So the avg data processing pipeline now has some ingest service that writes data to kafka and some processing pipeline reading of kafka and doing processing. So on ops side you have to maintain a lot of pieces including Kafka, Zookeeper etc. And if we had integrated solution Elixir would be super compelling story for these types of projects.

Considering they sound like primarily a distributed configuration store with the ability to notify, why not just use the built in mnesia in that case, simple function wrappers can perform the notifications on every node and all.

/me trying to understand what they do as not heard of them before… ^.^

I don’t think Mnesia implements distributed consensus (etcd is using RAFT and Zookeeper ZAB(similar to paxos))

There are layers on top of it though. In addition there are a host of libraries in the erlang ecosystem that definitely do in a variety of ways for whatever would work best for you (Riak-Core is one such consensus library that I think works well standalone unlike the entirety of Riak).

/me has noticed even more unknown words like RAFT, ZAB, and paxos… >.>

Riak core at first look looks to be purpose built for Dynamodb style thingys.

Not sure how dynamodb does it, but riak_core works like:

• You set up an abstract space, these are basically ‘partitions’, more tends to be better until they get to hold too little. Your maximum number of nodes that can be joined is the partition count, so setting it to like 128 or so is usually overkill. The partitions form an indexable cyclic ring.
• There are ‘virtual nodes’, each holds a single partition. These virtual nodes are separated and distributed evenly around the number of actual nodes in your system (distributed to keep ‘nearby’ partitions in the ring to be on as many nodes as possible).
• The virtual nodes themselves are what do ‘work’. They receive and (optionally) respond to commands (not necessarily to the sender). Riak_Core I think comes with a default set of virtual nodes that act as a work queue so you can submit work to it like poolboy (the ‘command’ is just a function to execute in this case).
• Riak uses it to store key/value information in it, it uses a distributed hasher to uniquely define where the key belongs in the partition ring (this is all handled internally), it then acquires the virtual node that matches that location and sends the command to store it it in that one as well as in the adjacent few before and after (depending on the quorum configuration you have it set to) and those virtual nodes will do whatever you wanted with the key/value, such as store it, retrieve it, mutate it, etc…
• You can however do whatever you wish in a virtual node, from being a processing array to store/fetch data to whatever.
• Riak_core has a riak_core_bucket thing built in that acts as a simple namespace with configurable properties, good for storage of information about the ring itself or small amounts of other data (say configuration data).
• Actual nodes that host virtual nodes can be taken up and down as wished, the virtual nodes automatically migrate to actual nodes that have the capabilities they need.

The example of use is of course Riak itself, say you set a quorum of 3/2, so 3 nodes have to verify a data write before a write is successful (the write message returns) and 2 nodes have to verify a data read (and they must match, otherwise all 3 will be checked and the 3rd will be temporarily taken offline as it performs a full resync, this will not affect the uptime of anything and is usually pretty close to instant anyway). As storage happens the information replicates around the ring until they are all in synch (but certain virtual nodes basically ‘own’ certain keys). This means that processing is distributed around the node while knowing that the value you ask for is written correctly as well as is read up-to-date since they get serialized through the quorum processes.

There are a lot of other aspects of it and this is a high level overview, but it is quite nice. I’ve used Riak in the past for quite a number of things (though nowadays, to be honest, I’d just use PostgreSQL, it even has notification on change functionality and all).

EDIT: The intro post of Riak Core:

One major difference is that Zookeeper is CP while Riak is AP.

Riak has the best tools for dealing with eventual consistency that I’ve ever seen in a product. Things like sibling detection and resolution. But still, your application needs to be consistency tolerant and be aware of the sibling problem. Many aren’t and for them a CP solution that is A as much as possible makes sense.