OK let’s not call it clone 
Ok. But then I don’t know what we’re building, or how you’d “beat” Kafka with it. Kafka comes with a bunch of known quantities. I was hoping someone would explain how building Kafka in Elixir would somehow yield a superior Kafka product.
If we’re talking about a completely different product that is not similar to Kafka but that might seek to obsolete it some day, that’s probably a different thread. Or at least different from what I thought the OP was asking about.
RabbitMQ might “beat” ActiveMQ and we can discuss why. ZeroMQ doesn’t beat RabbitMQ/ActiveMQ. They aren’t similar enough things.
1 Like
I think calling interface is a separate concern. I don’t think that systems not written for BEAM would use this.
Can we beat Kafka if we build it in Elixir?
Yes.
As others have mentioned, Kafka as implemented is a very narrow, well-engineered tool. The only way to “beat” it would be to add value to their approach fundamentally.
As mentioned, ESB approaches bring scaleability issues. This is inherent to the immutable, distributed transaction log approach, event stores/sourcing, as well as more conventional rdbms as well (which rely heavily on transactional logs under the covers). And more recently, blockchain “dbs” are running into the same issues as well, being solved by fracturing through both hard forks and simply new blockchains. I say inherent because the DB and its configuration are separate data structures, which does not evolve and distribute easily.
IPFS looks very promising in it’s approach to this, but it is written in go and js. But this is precisely the scaleability issue that ibGib (src) addresses with it’s design, and it is written in Elixir (though isomorphic engines in other languages would be ultimately implemented just as IPFS is attempting).
ibGib incorporates a tree-shaking capability, which is similar to creating a stream projection in the event stream paradigm. The difference is that each record contains its own self-contained projection, so there is no single source of truth. This includes data, as well as any metadata, e.g. configuration. And the integrity is maintained via the merkle DAG (although as I’ve mentioned elsewhere it is both cyclic and acyclic).
There are other hard problems addressed in the design, largely related to growing identity and integrity beyond clusters. And this scaleability issue is only one of the multiple aspects that enable it to be the next architectural evolution for the distributed programming model.
Btw, I’m at 530+ bicycle miles for my path for ibGib now, currently in Muleshoe - almost out of Texas! 
3 Likes
Maybe little of topic but there is alternative to Kafka - NATS (written initially in rubby then in GO)
It seems there is elixir client also.
We are a Cloud Native Computing Foundation member project. Members of the Synadia team created the NATS Server (written in Go), NATS Streaming, and clients written in Python, Ruby, Node.js, Elixir, Java, NGINX, C, and C#. The community has contributed a growing list of libraries, including Arduino, Rust, Lua, PHP, Perl, and more.
2 Likes
Is high quality project Derek Collison who created it was one of the creators of the message bus product at TIBCO, than was doing similar projects at Google for a while so he literally has a ton of experience building solutions in this area.
2 Likes
Just ran across this (it’s in Erlang): GitHub - erleans/vonnegut
Vonnegut is a append-only log that follows the file format and API of Kafka 1.0. The server can be run standalone, with 1 or more chains each with 1 or more replicas, or as part of another Erlang release which can talk to it directly.
5 Likes
FYI, I recently joined the IPFS core team but much prefer Elixir – suggesting a port is obviously not something I’ll do any time soon but I am interested in hacking on it at some point just to see how it does. Anyway, just thought you might be interested.
5 Likes
Thank you for the response! I have kept up with many many IPFS online meetings (core, js, workgroups, etc.) for the past several years, and it still looks very interesting. I hope you have a good time on their core team, as they seem like a good group of folks. I always wondered though why Juan Benet never was on them 
, though David himself seems quite brilliant.
For future reference, regardless of which language implementation you’re hacking on, I would recommend converting your source code VCS (and Good Lord, the meta documentation, tracking, etc.) to IPFS itself instead of the patched approach with the waffle board and git! This would allow for, among (many) other things, more fine-grained porting of code among languages and the ability to create meta-compilers post hoc (think version control at the function level instead of repo level).
Looking to the Github repo it looks like you have paused your idea of building the distributed log or are you just happy as it is in this moment?
