Can we beat Kafka if we build it in Elixir?

I can’t answer your question, but bear in mind that it’s possible that kafka is actually more optimized for what it does than the Erlang VM. The Erlang VM is optimized for server architectures and easy concurrency, but it’s possible to do better if you have a very specific task.

It would be a duplication of effort. Why not build something new.

It would be super useful even if it does not perform as well as Kafka just for ops simplicity

I know they are not absolutely equivalent, but if you want to see something similar to Kafka, that is written in Erlang, look at RabbitMQ.

Pretty different use cases between RabbitMQ and Kafka

I’m interested to know what you mean when you say, “all the features what the kafka is providing are already in Erlang”. Certainly all of the building blocks are there. But Kafka is a fairly narrow tool, albeit a robust one. I’m not sure we have something that is that focused in elixir.

Along these same lines I’ve been toying with the idea of using our raft implementation to build out a distributed log. I use Kafka pretty heavily at work but I think that could be a really useful thing to have available in elixir. But it shouldn’t be understated the amount of work and engineering that has gone into making Kafka as robust as it is.

I think it is about scale and how many people right now are contributing and using Kafka.
For small scale and simplicity suppose you don’t need Kafka and Erllang/Elixir is enough.

For big scale, multi regions Apache Kafka could be better choose.
But to maintain Kafka you will need separate team

Forget about building something … for the moment. Are there any ideas inside Kafka that are applicable to systems built in the BEAM/OTP ecosystem? I suspect there might be. Kafka in a Nutshell

On that note: ThoughtWorks: Recreating ESB antipatterns with Kafka

Kafka is becoming very popular as a messaging solution, and along with it, Kafka Streams is at the forefront of the wave of interest in streaming architectures. Unfortunately, as they start to embed Kafka at the heart of their data and application platforms, we’re seeing some organizations recreating ESB antipatterns with Kafka by centralizing the Kafka ecosystem components — such as connectors and stream processors — instead of allowing these components to live with product or service teams. This reminds us of seriously problematic ESB antipatterns, where more and more logic, orchestration and transformation were thrust into a centrally managed ESB, creating a significant dependency on a centralized team. We’re calling this out to dissuade further implementations of this flawed pattern.

Also culturally speaking successful JVM-based projects have a tendency towards bloat as they mature, while sometimes not minimizing their (inter-)dependencies. I have no idea if that is the case for Kafka - but if it is, it might be getting to the point of moving from “just use Kafka” to “you don’t need Kafka for that”.

The Handling Failure section of the above article reminded me of the Consensus and Leader Election section of your talk (good one, BTW).

But it shouldn’t be understated the amount of work and engineering that has gone into making Kafka as robust as it is.

To some degree that engineering could be more valuable than the product itself. But you would have a better idea of whether Kafka may have gotten a bit “bulky” for some business use cases and whether there are instances where a lighter weight alternative may be a better fit, provided the organization isn’t already using Kafka for some other, legitimate reason.

I’m not sure whether there is a clear point at which the benefits of adopting Kafka outweight the costs. Clarifying that in itself could be valuable. Naturally there already are opinions “to adopt Kafka before you need it” so you’re familiar with it, once you do need it.

In the short term it probably makes more sense to focus on one single, excellent Kafka client library for Elixir so that organizations already using Kafka don’t reject Elixir out of hand because of suboptimal integration with Kafka (seems some complaints have more to do with the available Kafka client libraries than Kafka itself).

Which Kafka lib are you using? How stable is it?

To me most useful use of Kafka is a buffer that can absorb events in data pipeline that you can process async. It allows you to handle spikes well (up to a point). I am not a big fan of ESB but if I had to build a project around ESB I’d rather use NATS.

In some circles an Event Service Bus in itself is considered an anti-pattern.
Why you don’t need an Enterprise Service Bus (ESB);