Klife - A Kafka client with performance gains over 10x

Schultzer · December 9, 2025, 6:42pm

I’ll be giving this library a go, I’ll let you know if I experience any issues. Great to see performance being prioritized, which is essential for a well engineered piece of software!

dimitarvp · December 9, 2025, 6:44pm

Has this library been tried with Redpanda?

oliveiragahenrique · December 9, 2025, 7:08pm

Great to hear that! Let me know if you find any issues.

I’m working on a new release with consumer features supporting the new rebalance protocol available since Kafka 4.0 ( Confluence Mobile - Apache Software Foundation ) and with that a 1.0 release.

For the 1.0 regarding the producer, the only breaking change I have planned for it is the default partitioner algorithm from phash to murmur in order to keep it consistent with other Kafka client libraries.

If you have other suggestions let me know so I can consider them for the 1.0

oliveiragahenrique · December 9, 2025, 7:13pm

Early versions have been tested with redpanda, but I’m not sure if everything still works. AFAICT it should work.

The library tests run against a docker compose Kafka setup, one way to check would be setting up a docker compose with redpanda and run the tests!

Schultzer · December 9, 2025, 7:18pm

I got no issues with breaking changes, right now I just need consumers.

oliveiragahenrique · December 9, 2025, 7:25pm

Which version of Kafka are you running? I have a working consumer to be released soon (early january), but it will only work with Kafka version >= 4.0

Schultzer · December 9, 2025, 7:30pm

Bummer, we’re using Kafka 3.8

manhvu · December 10, 2025, 2:01am

Redpanda is nice, it’s performance is better and consumption less resource Kafka (around 1/3 resources compare with Kafka in our case). It provides Kafka APIs compatible. I have build SoC (Security Operations Center) with Redpanda before and I love that.

I’m finding a Kafka client with good performance for bus message between regions for Elixir ecosystem. Nice to hear about Klife lib.

cjbottaro · December 10, 2025, 9:42pm

We use NATS Jetstream for the exact same reason. Having a consumer be a first class object that lives on the server is really nice. We don’t need the raw throughput of Kafka, so Jetstream is awesome.

I wrote everything from scratch. From the low level NATS client, to higher level Jetstream client, to pull/push consumer abstractions. None of it is open source cuz I’m an ahole, huzzah… (I find open source maintainers to be extremely admirable and we’re all building on their giant shoulders… I just can’t do it well myself)