What Kafka client do you recommend?

We’d like to start using RedPanda at work and I am hunting for a Kafka client library.
There are three options I’ve seen so far:

  1. kafka_ex. It looks like it is in a transition period between 0.x to 1.x. There is KafkaEx.New.KafkaExAPI.
  2. brod. Made by Klarna, looks stable
  3. kaffee. A wrapper around brod that in the README claims to be experimental.

I don’t mind using experimental or unstable code. I could potentially contribute but is there someone who used at least two of them and can compare? Do they differ in architecture? Is one of them easier to use? Do they make different trade-offs?

While searching the forum, I saw @keathley shared a lot about Kafka usage. Thanks for the slides from your presentation about it! Is it available somewhere as a video? I hope you don’t mind the ping :innocent:

1 Like

I prefer to use Brod, but Brod is kinda weird. I ended up writing my own wrapper at B/R that never got open sourced. You want to avoid a lot of the wrappers that use GenStage because they don’t handle partitions correctly and will allow messages to be processed out of order. Obviously, this only matters if you rely on kafka’s partition ordering (we did). But overall Brod will do what you need and have the fewest surprises. Build your own wrapper once you know what features or improvements you need for your use case.

4 Likes