Which Kafka lib are you using? How stable is it?

I’m currently using GitHub - kafkaex/kafka_ex: Kafka client library for Elixir · GitHub as it seems to be the most mature solution

Works fine for producing, but I haven’t gotten around to setting up consumers yet

We’re using both kafka_ex and Brod. Brod seems to be more complete, especially on the consumer side, and we noticed that produce calls also complete quite a bit quicker (but that may be just a simple-to-fix bug, I don’t think it’s something wrong in the design of kafka_ex).

They both work, they have different designs, YMMV, etc.

Thanx for sharing, do you consider it to be production ready ? Any gotchas to watch out for?

I don’t know which library you refer to as “it”, but we’re running both in production, so that’s a simple question to answer: yes.

Do you have any opinions on the consequences of there differing designs?
Also how come you are using both libraries

We started just producing - kafka_ex was fine for that. Then we needed a consumer in a new service, kafka_ex didn’t back then (it does now) support consumer groups in the 0.9 style, so we switched to Brod. Recently, we added Kaffe to that as its batched consumer does what we want and it is minimal code (Kaffe is opinionated, we can live with its opinions, and then all you need is basically a single method). We may at some point in the near future do a formal shoot-out and decide to stick with one or the other though ;-).

I don’t have strong opinions on either one - it’s not a super big issue for us, switching between both should be relatively simple, and bigger fish are waiting to be fried. We don’t do crazy volumes, so pretty much anything that talks the protocol will work for us; that might change as well and then we’ll certainly start forming opinions.

Hi there,

Sorry to write to a topic that is more than two years old but I am pretty new to Elixir and wants to write an application that uses Kafka. I have found kafka_ex and kaffe, but do not know wich library to choose right now..

Kafka_ex seems to be changing to a new architecture … and Kaffe, well I have tried to setup a simple consumer example following the instructions with no success ( probably my fault ) , and there are no much examples out there.

Any recommendations regarding a Kafka Library in the 2020?

Thanks in advance.

I’m using brod, with thin wrapper brodex. I wish brod has built-in Elixir binding (like jose).

I checked kafka_ex again.. and KafkaEx 1.0 API seems good to me - but haven’t used kafka_ex.

BTW broadway_kafka uses brod as kafka library.

BroadwayKafka can subscribe as a consumer to one or more topics and process streams of records within the same consumer group. Communication is done through Kafka’s Consumer API using the :brod client.

I’m a heavy Kafka user, and I have been bouncing between Brod, KafkaEx, an BroadwayKafka for a long time.
Until now I have been using Brod, which was a bit abandoned until a month ago, when it was donated to the community by the company that started the project.
Now mainly zmstone and k32 are maintaining the library but they still have too much work to do. Meanwhile, Kafka_ex has also many PRs opened, and lot of work to be done. In both cases, I really think maintainers needs help (and I would love to at least send them a ko-fi).

My problem is that I currently need lz4 compression and Exactly once delivery, and neither of them are there yet. Also I’ve some issues with the consumer group balancer in brod, because our running Kafka version uses an old balancing protocol.

Recently I started looking at the Python, Ruby, Go and Js libraries. Official ones (from Confluent) depends on GitHub - confluentinc/librdkafka: The Apache Kafka C/C++ library · GitHub (C/C++) which has a lot of support.

Of course, there is also one for the BEAM (GitHub - silviucpp/erlkaf: Erlang kafka driver based on librdkafka · GitHub). I’ve used it in development environment, and until now, it works perfectly well. For example, I experimented consuming with a topic with 12 partitions, with 1 node, it automatically spawns 12 processes, I run another node and automatically balances 6 and 6 processes (That is handled by the Kafka protocol).
Someone could be worried by nifs but as erlkaf readme says: librdkafka driver is async, so it doesn’t block.

So I think, for now, I will be migrating to erlkafka. But I will keep looking for brod and kafka_ex, hoping they get more support.

Lz4 is now available in Kafka_protocol (brod)