I’m a heavy Kafka user, and I have been bouncing between Brod, KafkaEx, an BroadwayKafka for a long time.
Until now I have been using Brod, which was a bit abandoned until a month ago, when it was donated to the community by the company that started the project.
Now mainly zmstone and k32 are maintaining the library but they still have too much work to do. Meanwhile, Kafka_ex has also many PRs opened, and lot of work to be done. In both cases, I really think maintainers needs help (and I would love to at least send them a ko-fi).

My problem is that I currently need lz4 compression and Exactly once delivery, and neither of them are there yet. Also I’ve some issues with the consumer group balancer in brod, because our running Kafka version uses an old balancing protocol.

Recently I started looking at the Python, Ruby, Go and Js libraries. Official ones (from Confluent) depends on GitHub - confluentinc/librdkafka: The Apache Kafka C/C++ library · GitHub (C/C++) which has a lot of support.

Of course, there is also one for the BEAM (GitHub - silviucpp/erlkaf: Erlang kafka driver based on librdkafka · GitHub). I’ve used it in development environment, and until now, it works perfectly well. For example, I experimented consuming with a topic with 12 partitions, with 1 node, it automatically spawns 12 processes, I run another node and automatically balances 6 and 6 processes (That is handled by the Kafka protocol).
Someone could be worried by nifs but as erlkaf readme says: librdkafka driver is async, so it doesn’t block.

So I think, for now, I will be migrating to erlkafka. But I will keep looking for brod and kafka_ex, hoping they get more support.

I don’t know which library you refer to as “it”, but we’re running both in production, so that’s a simple question to answer: yes.

We started just producing - kafka_ex was fine for that. Then we needed a consumer in a new service, kafka_ex didn’t back then (it does now) support consumer groups in the 0.9 style, so we switched to Brod. Recently, we added Kaffe to that as its batched consumer does what we want and it is minimal code (Kaffe is opinionated, we can live with its opinions, and then all you need is basically a single method). We may at some point in the near future do a formal shoot-out and decide to stick with one or the other though ;-).

I don’t have strong opinions on either one - it’s not a super big issue for us, switching between both should be relatively simple, and bigger fish are waiting to be fried. We don’t do crazy volumes, so pretty much anything that talks the protocol will work for us; that might change as well and then we’ll certainly start forming opinions.