Forget about building something … for the moment. Are there any ideas inside Kafka that are applicable to systems built in the BEAM/OTP ecosystem? I suspect there might be. Kafka in a Nutshell

On that note: ThoughtWorks: Recreating ESB antipatterns with Kafka

Kafka is becoming very popular as a messaging solution, and along with it, Kafka Streams is at the forefront of the wave of interest in streaming architectures. Unfortunately, as they start to embed Kafka at the heart of their data and application platforms, we’re seeing some organizations recreating ESB antipatterns with Kafka by centralizing the Kafka ecosystem components — such as connectors and stream processors — instead of allowing these components to live with product or service teams. This reminds us of seriously problematic ESB antipatterns, where more and more logic, orchestration and transformation were thrust into a centrally managed ESB, creating a significant dependency on a centralized team. We’re calling this out to dissuade further implementations of this flawed pattern.

Also culturally speaking successful JVM-based projects have a tendency towards bloat as they mature, while sometimes not minimizing their (inter-)dependencies. I have no idea if that is the case for Kafka - but if it is, it might be getting to the point of moving from “just use Kafka” to “you don’t need Kafka for that”.

The Handling Failure section of the above article reminded me of the Consensus and Leader Election section of your talk (good one, BTW).

But it shouldn’t be understated the amount of work and engineering that has gone into making Kafka as robust as it is.

To some degree that engineering could be more valuable than the product itself. But you would have a better idea of whether Kafka may have gotten a bit “bulky” for some business use cases and whether there are instances where a lighter weight alternative may be a better fit, provided the organization isn’t already using Kafka for some other, legitimate reason.

I’m not sure whether there is a clear point at which the benefits of adopting Kafka outweight the costs. Clarifying that in itself could be valuable. Naturally there already are opinions “to adopt Kafka before you need it” so you’re familiar with it, once you do need it.

In the short term it probably makes more sense to focus on one single, excellent Kafka client library for Elixir so that organizations already using Kafka don’t reject Elixir out of hand because of suboptimal integration with Kafka (seems some complaints have more to do with the available Kafka client libraries than Kafka itself).

Which Kafka lib are you using? How stable is it?

The way we use kafka is as an immutable log of facts. Things happen, those things generate facts, and the facts are added to the log. Our usage treats kafka much more like a database then anything else. Consumers can read from them in a demand driven fashion. None of these ideas are novel to our company; lots of people use kafka for this. But this architecture does lets us create decoupled applications using facts from different parts of our company.

The benefit to having a solution in elixir for me personally would be operational overhead (I know exactly enough about tuning jvms and operating kafka and zookeeper to get myself into trouble) and ease of adoption. Trying to rival kafka’s scalability and robustness is going to be hard to do and, frankly, most people are probably better off just using kafka.

Raft is already a distributed, consistent log so its a good choice as a primitive for a higher level log abstraction. This blog series explains more about it: Building a Distributed Log from Scratch, Part 1: Storage Mechanics – Brave New Geek. If people are interested in working on something like that then I’d be happy to chat with them.

We’re using brod and are really happy with it.

It would be super useful even if it does not perform as well as Kafka just for ops simplicity