Announcing Klife: A High-Performance Kafka Client for Elixir
I’m thrilled to share the next chapter in a journey that began a couple of years ago—introducing Klife, a Kafka client for Elixir, designed with a strong focus on performance and efficiency.
Currently, Klife supports producer functionalities, and I’m excited to start designing the consumer system later this year, though it may take some time before it’s ready for release.
The journey began with Klife Protocol, where I explored reimplementing Kafka’s protocol entirely in Elixir. This process opened up new possibilities to optimize the protocol itself, and I found ways to achieve more with less overhead.
Key to Klife’s Performance Gains: Batching
One of Klife’s primary performance improvements is achieved through batching. By bundling data destined for the same broker into a single TCP request, Klife significantly enhances performance, especially in high-throughput scenarios.
In benchmarking against two fantastic, well-established libraries, brod and kafka_ex, Klife achieved up to 15x higher throughput for producing messages. You can find details about the benchmarking setup and results in the project’s documentation.
Key features
Klife includes several features designed to improve both performance and ease of use:
- Efficient Batching: Batches data to the same broker in a single TCP request per producer.
- Minimal Resource Usage: Only one connection per broker for each client, optimizing resource usage.
- Synchronous and Asynchronous Produce Options: Synchronous produces return the offset, while asynchronous produces support callbacks.
- Batch Produce API: Allows batching for multiple topics and partitions.
- Automatic Cluster and Metadata Management: Automatically adapts to changes in cluster topology and metadata.
- Testing Utilities: Includes helper functions for testing against a real broker without complex mocking.
- Simple Configuration: Streamlined setup for straightforward use.
- Comprehensive Documentation: Includes examples and explanations of trade-offs.
- Custom Partitioner per Topic: Configurable partitioning for each topic.
- Transactional Support: Supports transactions in an Ecto-like style.
- SASL Authentication: Currently supports plain authentication.
- Protocol Compatibility: Supports recent protocol versions, with forward compatibility in mind.
Looking ahead
I hope Klife proves useful for those in the community, especially for use cases where producer performance is crucial. While it’s currently producer-only, the efficiency and resource gains might be valuable for some applications.
I’d love to hear any feedback, ideas, or issues you may encounter while trying it out. Thank you to everyone in the Elixir and Kafka communities who has inspired and supported this journey—Klife wouldn’t exist without you!
hexdocs: README — Klife v0.1.0