Has this library been tried with Redpanda?
dimitarvp
oliveiragahenrique
Great to hear that! Let me know if you find any issues.
I’m working on a new release with consumer features supporting the new rebalance protocol available since Kafka 4.0 ( Confluence Mobile - Apache Software Foundation ) and with that a 1.0 release.
For the 1.0 regarding the producer, the only breaking change I have planned for it is the default partitioner algorithm from phash to murmur in order to keep it consistent with other Kafka client libraries.
If you have other suggestions let me know so I can consider them for the 1.0
0
oliveiragahenrique
Early versions have been tested with redpanda, but I’m not sure if everything still works. AFAICT it should work.
The library tests run against a docker compose Kafka setup, one way to check would be setting up a docker compose with redpanda and run the tests!
1
Schultzer
oliveiragahenrique
Which version of Kafka are you running? I have a working consumer to be released soon (early january), but it will only work with Kafka version >= 4.0
0
Schultzer
manhvu
Redpanda is nice, it’s performance is better and consumption less resource Kafka (around 1/3 resources compare with Kafka in our case). It provides Kafka APIs compatible. I have build SoC (Security Operations Center) with Redpanda before and I love that.
I’m finding a Kafka client with good performance for bus message between regions for Elixir ecosystem. Nice to hear about Klife lib.
1
cjbottaro
We use NATS Jetstream for the exact same reason. Having a consumer be a first class object that lives on the server is really nice. We don’t need the raw throughput of Kafka, so Jetstream is awesome.
I wrote everything from scratch. From the low level NATS client, to higher level Jetstream client, to pull/push consumer abstractions. None of it is open source cuz I’m an ahole, huzzah… 
(I find open source maintainers to be extremely admirable and we’re all building on their giant shoulders… I just can’t do it well myself)
1
oliveiragahenrique
And people say programmers can’t estimate! 
I’m very glad to share that the first official release of Klife’s consumer features is out. It supports Kafka >= 4.0 and already includes some of the newer capabilities from KIP-848, which should bring meaningful improvements in performance and consumer stability.
If this is useful for your use case, give it a try! Feedback is very welcome.
Thanks!
8
Schultzer
I will give it a spin, as we have finally upgraded to Kafka 4.0
FWIW I implemented a consumer with brod but that was a miserable experience as every time the Kafka Broker rebalance or what not the client spam useless errors about socket closure that I cannot do anything about, hopefully your work will just do the right thing and only error when the user has and can meaningfully address an issue the library has absolutely no way to address.
2