When you want to access Kafka you go to ZK and it points you direct to node.
In this way you can add nodes for scaling and client does not need to know about this …
The same works Solr Welcome to Apache Solr - Apache Solr
But ZK can be also problematic
I think other’s have also pointed out:
https://github.com/bitwalker/libcluster
https://github.com/bitwalker/swarm
Swarm uses libcluster under the covers (it seems nice for stand-alone node discovery). You might also roll in rafted_value on top of one or both of these to build your solution.
You’ll just have to ask yourself what kind of solution you’re trying to bake into your app cluster. Is it distributed queueing with individual consumer offset (like Kafka)? Is it stateful actors? Is it stateless workers?
For distributed queueing with individual consumer offsets, Kafka is pretty popular. For stateless workers registered across a cluster, swam seems like it solves that.
It could be interesting to marry stateful persistent actors and an intelligent global registry. This appears to be missing in the Elixir world, although there may be some distributed Ecto project that I’m unaware of. The characteristics would be:
This ought to be a pretty idiomatic strategy for “stateful elixir”, but it’s not easy to get right. It depends on a single authoritative writer and those aren’t easy to deterministically manage in a distributed system. Process crashes, node crashes, and netsplits need to be considered. If you want to increase read throughput you can add replicas, but this further complicates things. If you want to get really fancy you could configure dynamic support for ETS or FastGlobal reads.
A lot of the complexity is in the distribution and coordination methodology. Let’s say you need need to track 10 million persistent actors. That’s too big for gossip. How about RAFT? I don’t know. What happens when the cluster detects a node has gone down? Do you have the ability to hand off work to another node immediately and deterministically? If the original node comes back, does it have the ability to take back over? Do nodes benefit from local state caching, or do nodes that fail need to regenerate state from a remote master, possibly flooding that system with millions of requests as they come back up?
This would be a PC/EC libriak that seeks to keep A has high as possible. You could probably tune this to be PC/EL if you wanted to support additional replicas and could track exactly what the latest state version of each actor was.
That would be a project I’d be interested in playing with.
I think some of the goals are shared by Erleans - it strives to implement a similar ecosystem of “virtual actors” to that available in .NET through Microsoft’s Orleans framework. It also uses lasp underneath and CRDTs to guarantee evental consistency. It’s definitely an interesting project to look out for.
As has already been pointed out by many here, there are many, many great libraries / frameworks that provide the underpinnings for these things for Elixir already. RiakCore, RaftFleet, libCluster, mNesia, etc. etc.
But I think what is being missed a bit is the “batteries included” part of this. Yes, you can build pretty much any distributed system you need from those lego blocks but etcd/zookeeper are very specific applications: distributed configuration data.
It would not be hard to build something that is a config store with distribution, pub/sub on changes, initial load on application start, etc. but AFAIK that specific application does not exist. The lego pieces are still in the bucket waiting to be put together, and this pattern results in many higher-level applications (like distributed config management) left as an exercise to the individual developer rather than being pulled together into reusable projects.
Hopefully over time more of these sorts of projects will hit hex.pm, so Elixir can have more “finished pieces” to use alongside the underlying toolboxes.
The answer is “it depends”. Kafka does a lot of batching 
Just for fun, I wrote a Raft implementation as well (https://github.com/cdegroot/palapa/tree/master/apps/erix); it’s not too hard, and I religiously TDD’d by basically converting the specifications from the paper into tests, and I might this year or next year give a fun demo at some meetup or conference about it…
But would I use it in any production-like case? Nope. Ditto for most of the other Elixir/Erlang based stuff out there, including riak_core. Look at the stream of work that goes into Zookeeper, Consul and etcd; getting this right is hard, and I think that if you want industrial strength, you have to leverage one of these large projects. Using your own or having something maintained by the tiny Elixir/Erlang community is most likely going to end up in tears.
Yes it would. Do a round through the issue lists of the aforementioned “big three” products.
The counterpoint to the size of the community is the size and mission-criticality of the products some of these components are used in, and the testing they get.
You mentioned raft; we’re using am Elixir raft implementation here at work that is written, used and maintained by a reasonably large Japanese company. They commit regularly, and they use the libraries themselves. And that’s a very small example compared to e.g. riak libraries being used for core functionality at companies like Bet365.
On the flip side the Javascript community is absolutely immensely huge, and the quality of projects that come from there is often scary-low and the lifespan way shorter than one would hope for.
Size of community alone is really not enough to make decisions based on. It can help, but it needs to be put into the context of the technology and commitments to it.
I’m very familiar with them, but thanks! 
The original poster mentioned etcd and zookeeper, and having something that provided that specific functionality out-of-the-box would really not be a huge effort within the existing Elixir ecosystem as a lot of the really hard work (distribution, global naming, CRDTs and other EC helpers ..) has already been done either in the core libraries or widely used add-on libraries that have been proven stable over time, under load, and at scale.
So why are these things such monumental efforts for other communities? Because they don’t start with the tools Elixir benefits from when it comes to distributed computing. They have to build a lot more of the primitives from the ground up which we get to just pick off the ol’ hex.pm tree, or which are baked into the language design itself.
Well tiny Erlang community already maintains pretty much the most advanced commercially viable VM for distributed systems. CoreOS (or whatever the current name is) was a tiny company until few days ago when it was acquired by Red Hat etcd obviously benefited greatly from being part of k8s efforts though. This def. would take concentrated effort from larger players that have use cases for something like this e.g the likes of Toyota, bet365, Erlang Solutions, Ericsson, Pinterest etc. or alternatively by a startup that for example was building event processing PaaS.
I’d grant you that for ZK ;-). etcd and Nomad are written in Golang, and while our respective communities may disagree about a lot, I think we can agree on the fact that Golang was created with distribution in mind.
My experience with these sorts of systems are that they’re 20% use case, 80% edge case. https://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/ is a good example.
You fix an edge case, and another edge case appears on someone else’s system. Frankly, the couple of times I wrote low level networking stuff I had the feeling it was a fractal all the way down and it ended up costing me way more time than I cared for. I guess that if you have limited deployment and stay in the sweet spot, things will be relatively easy; something that works for all of the Erlang/Elixir community, I’m less sure. My take so far has been to just use ZK for the one or two use cases where we need network-wide singletons; pretty much everything else we do uses Kafka and thus relies on its consumer group functionality to divide up work, hence indirectly using ZK as well. One less headache for me. YMMV.
Most Javascript projects deserve a lifespan of zero. Starting with the language itself. As far as I have any hopes for the GUI platform called the web browser, it’s that webassembly takes off and proper languages can be used as first-class citizens
Yet it has none of the features that make creating and monitoring work and state on another node possible, much less a way to actually connect two runtimes. Does it even have a primitive for referring to threads?