Erlang/Elixir native Etcd, Zookeeper alternative

I think some of the goals are shared by Erleans - it strives to implement a similar ecosystem of “virtual actors” to that available in .NET through Microsoft’s Orleans framework. It also uses lasp underneath and CRDTs to guarantee evental consistency. It’s definitely an interesting project to look out for.

I think other’s have also pointed out:

https://github.com/bitwalker/libcluster
https://github.com/bitwalker/swarm

Swarm uses libcluster under the covers (it seems nice for stand-alone node discovery). You might also roll in rafted_value on top of one or both of these to build your solution.

You’ll just have to ask yourself what kind of solution you’re trying to bake into your app cluster. Is it distributed queueing with individual consumer offset (like Kafka)? Is it stateful actors? Is it stateless workers?

For distributed queueing with individual consumer offsets, Kafka is pretty popular. For stateless workers registered across a cluster, swam seems like it solves that.

It could be interesting to marry stateful persistent actors and an intelligent global registry. This appears to be missing in the Elixir world, although there may be some distributed Ecto project that I’m unaware of. The characteristics would be:

• Actors/processes have state
• They delegate to backends for save/restore functionality
• They have a formal API for interaction (GenServer vs Agent – although a persistent agent wrapper would be ok for some projects)
• They are sharded across your cluster
• They are communicated to in a location transparent way
• Node failures are detected immediately
• Network splits are detected immediately and fall back to a well defined “mode”
• Node rebalancing and recovery is fast and deterministic

This ought to be a pretty idiomatic strategy for “stateful elixir”, but it’s not easy to get right. It depends on a single authoritative writer and those aren’t easy to deterministically manage in a distributed system. Process crashes, node crashes, and netsplits need to be considered. If you want to increase read throughput you can add replicas, but this further complicates things. If you want to get really fancy you could configure dynamic support for ETS or FastGlobal reads.

A lot of the complexity is in the distribution and coordination methodology. Let’s say you need need to track 10 million persistent actors. That’s too big for gossip. How about RAFT? I don’t know. What happens when the cluster detects a node has gone down? Do you have the ability to hand off work to another node immediately and deterministically? If the original node comes back, does it have the ability to take back over? Do nodes benefit from local state caching, or do nodes that fail need to regenerate state from a remote master, possibly flooding that system with millions of requests as they come back up?

This would be a PC/EC libriak that seeks to keep A has high as possible. You could probably tune this to be PC/EL if you wanted to support additional replicas and could track exactly what the latest state version of each actor was.

That would be a project I’d be interested in playing with.

Yep looks interesting thank you

When you want to access Kafka you go to ZK and it points you direct to node.
In this way you can add nodes for scaling and client does not need to know about this …
The same works Solr Welcome to Apache Solr - Apache Solr

But ZK can be also problematic

If you want RAFT internal to your cluster, this looks pretty good:
https://github.com/skirino/rafted_value

No argument there although I highly doubt anyone is in a hurry to put a burden on their team and try to implement RAFT or Paxos themselves vs using ZK or Etcd.

It’s not about being data intensive, it’s about being coordination intensive. The more you have to coordinate the more coordination work you have to do. If your use case requires the serially deterministic coordination (PC/EC) that these products say they support, then your app is at the mercy of these products before/after/maybe during the heavy lifting that its doing.

It’s similar to what happens to the Erlang scheduler when you sprinkle in NIFs. If those NIFs are super fast, you’re ok. If they aren’t, your whole system can become destabilized.

And this coordination load can be proportional or independent of application write volume.

Ask Kafka if it would rather process 10x 1MB messages a second or 5 million 1 byte messages a second. I’m pretty sure the answer isn’t “it doesn’t matter”.

True but I am not aware of any product that stores anything data intensive in either etcd or ZK it is generally pretty much for config and meta data. Even Kafka itself has pretty low requirements since it’s basically writing to the log sequentially and is batching writes.

That’s good, I guess. But not super relevant. Of course, if you don’t actually drive ZK/etcd very hard, its performance doesn’t have as great an impact. But any Etcd/Zookeeper alternative (or Elixir implementation) would have to plan for heavy usage. And when that happens you end up with hard problems to solve. For both of these products, fast disk seems to be a requirement.

You’re also still back to the same quandary. These products need something special beyond what “normal” nodes are required to have. If you melt their functionality into your app/node, then your app/node also needs to assume this specialized characteristic, in addition to whatever product level requirements your distributed nodes have. That was all I was trying to point out in my original response. You can pass the buck around, but it still needs to be paid by something.

This is can have some volume but that was true for older versions of Kafka as far as I remember newer versions no longer store Consumer offset in ZK.