The issues usually aren’t performance but feature support
You are absolutelly right about performance not being the main issue for now!
As for features, most of the key gaps are on the consumer side, and I’m still evaluating the best approach there. I believe we can leverage BEAM’s distribution model to address certain issues uniquely—particularly rebalancing.
I have a few ideas to explore further before committing to a specific path, which may take some time. I’m estimating around six months to solidify the approach and potentially another year and a half to reach a releasable state, depending on my time availability.
There is some big changes comming for kafka such as:
- KIP-848: The Next Generation of the Consumer Rebalance Protocol - Apache Kafka - Apache Software Foundation
- KIP-932: Queues for Kafka - Apache Kafka - Apache Software Foundation
I plan to incorporate support for these from the start, leveraging the clean-slate approach I’m taking with Klife.
In short, I agree that feature support is the main pain point. For the consumer side, I can only speak to plans at this stage since it’s still in development.
For the current producer, though, I believe it’s in strong shape, with support for batch production, synchronous and asynchronous modes, idempotency with EOS, transactions, custom partitioning, and extensive performance optimizations. If there’s any producer feature you find missing, please let me know so we can look into it!
I’d love for us to have a great and robust kafka client, but at the same time I also believe that it’s too big for any one person to tackle.
I think it may be true, because Kafka is indeed a constantly evolving platform that relies on clients to take on a lot of responsibilities.
This was the main reasons I decided to build this project entirely from scratch, starting with a full protocol rewrite. To keep up with Kafka’s rapid development pace sustainably, especially with limited resources (time and money), it’s essential to have a complete understanding of the stack, end to end.
One project that’s been a big source of inspiration is Franz-go from Golang. It’s an impressive Kafka client that quickly integrates a wide range of KIPs, sometimes even outpacing the official Java client—all while being 99.9% maintained by a single developer.
I’m not sure if I’ll fully achieve this goal, but it’s definitely worth a shot—and the knowledge gained along the way has already been rewarding!
