Forget about building something … for the moment. Are there any ideas inside Kafka that are applicable to systems built in the BEAM/OTP ecosystem? I suspect there might be. Kafka in a Nutshell
On that note: ThoughtWorks: Recreating ESB antipatterns with Kafka
Kafka is becoming very popular as a messaging solution, and along with it, Kafka Streams is at the forefront of the wave of interest in streaming architectures. Unfortunately, as they start to embed Kafka at the heart of their data and application platforms, we’re seeing some organizations recreating ESB antipatterns with Kafka by centralizing the Kafka ecosystem components — such as connectors and stream processors — instead of allowing these components to live with product or service teams. This reminds us of seriously problematic ESB antipatterns, where more and more logic, orchestration and transformation were thrust into a centrally managed ESB, creating a significant dependency on a centralized team. We’re calling this out to dissuade further implementations of this flawed pattern.
Also culturally speaking successful JVM-based projects have a tendency towards bloat as they mature, while sometimes not minimizing their (inter-)dependencies. I have no idea if that is the case for Kafka - but if it is, it might be getting to the point of moving from “just use Kafka” to “you don’t need Kafka for that”.
The Handling Failure section of the above article reminded me of the Consensus and Leader Election section of your talk (good one, BTW).
But it shouldn’t be understated the amount of work and engineering that has gone into making Kafka as robust as it is.
To some degree that engineering could be more valuable than the product itself. But you would have a better idea of whether Kafka may have gotten a bit “bulky” for some business use cases and whether there are instances where a lighter weight alternative may be a better fit, provided the organization isn’t already using Kafka for some other, legitimate reason.
I’m not sure whether there is a clear point at which the benefits of adopting Kafka outweight the costs. Clarifying that in itself could be valuable. Naturally there already are opinions “to adopt Kafka before you need it” so you’re familiar with it, once you do need it.
In the short term it probably makes more sense to focus on one single, excellent Kafka client library for Elixir so that organizations already using Kafka don’t reject Elixir out of hand because of suboptimal integration with Kafka (seems some complaints have more to do with the available Kafka client libraries than Kafka itself).
