I don’t think so you can compare Kafka to ESB. ESB has complex logic inside itself where Kafka is just simple and append only transaction log. The business logic is outside Kafka in clients / producers of events.
Kafka is mostly used for stream processing / big data architecture
The way we use kafka is as an immutable log of facts. Things happen, those things generate facts, and the facts are added to the log. Our usage treats kafka much more like a database then anything else. Consumers can read from them in a demand driven fashion. None of these ideas are novel to our company; lots of people use kafka for this. But this architecture does lets us create decoupled applications using facts from different parts of our company.
The benefit to having a solution in elixir for me personally would be operational overhead (I know exactly enough about tuning jvms and operating kafka and zookeeper to get myself into trouble) and ease of adoption. Trying to rival kafka’s scalability and robustness is going to be hard to do and, frankly, most people are probably better off just using kafka.
Raft is already a distributed, consistent log so its a good choice as a primitive for a higher level log abstraction. This blog series explains more about it: Building a Distributed Log from Scratch, Part 1: Storage Mechanics – Brave New Geek. If people are interested in working on something like that then I’d be happy to chat with them.
We’re using brod and are really happy with it.
About operation I agree you need team or some external provider if you don’t wont to host itself 
I think somebody also felt operation pain and there is some attempt to implement it in GO
https://github.com/travisjeffery/jocko
Can we beat Kafka if we build it in Elixir
I think you can’t, but there is possibility to implement some compatible version in Elixr, Go , Rust which will be simple to maintain.
For example there is Casandra database on JVM and there is https://www.scylladb.com/ compatibile database written in C++.
Absolutely this. Better client libs would bring far more value to the Elixir ecosystem than building a Kafka clone in Elixir. This can be said for a good number of things.
Although I think the OP was really asking if a Kafka clone built in Elixir would be “better” than the current Kafka written in Java/Scala. Would it have better: speed, resource utilization, maintainability, support a wider range of clients, etc?
I think this boils down to an arms race between the JVM and BEAM. Assuming you’re still using 1st class client libs to access both implementations, what hypothetical differences would a JVM vs. BEAM Kafka have?
If we have an option of Elixir “clone” of Kafka that does not require ZK and integrates well with gen_stage it can be a killer app for people building data pipelines.
Any Kafka clone would still be an external process accessed via some client. That client ought to be able to integrate well with gen_stage. A better Kafka native client could integrate well with gen_stage as well. Kafka today supports Kafka streams for data pipelines. But this is just client magic on top of their existing product. A better Elixir client could expose a similar interface that leverages gen_stage.
Not having a ZK requirement might help operations run and maintain the new clone though. That could be a selling point. I like the memory characteristics of BEAM over JVM. Being able to take data off a NIC and write it directly to a file without intermediate mem copy, would also benefit a BEAM implementation. Although BEAM file IO is quite slow (at least for reads), so that’s a downside. BEAM IO is supposed to get some major love in OTP 21. I’m looking forward to seeing what they do there.