Subj. Will really appreciate learning about your experience using Elixir and or Erlang with Kafka.
I’m currently using https://github.com/kafkaex/kafka_ex as it seems to be the most mature solution
Works fine for producing, but I haven’t gotten around to setting up consumers yet
We’re using both kafka_ex and Brod. Brod seems to be more complete, especially on the consumer side, and we noticed that produce calls also complete quite a bit quicker (but that may be just a simple-to-fix bug, I don’t think it’s something wrong in the design of kafka_ex).
They both work, they have different designs, YMMV, etc.
Thanx for sharing, do you consider it to be production ready ? Any gotchas to watch out for?
I don’t know which library you refer to as “it”, but we’re running both in production, so that’s a simple question to answer: yes.
Do you have any opinions on the consequences of there differing designs?
Also how come you are using both libraries
We started just producing - kafka_ex was fine for that. Then we needed a consumer in a new service, kafka_ex didn’t back then (it does now) support consumer groups in the 0.9 style, so we switched to Brod. Recently, we added Kaffe to that as its batched consumer does what we want and it is minimal code (Kaffe is opinionated, we can live with its opinions, and then all you need is basically a single method). We may at some point in the near future do a formal shoot-out and decide to stick with one or the other though ;-).
I don’t have strong opinions on either one - it’s not a super big issue for us, switching between both should be relatively simple, and bigger fish are waiting to be fried. We don’t do crazy volumes, so pretty much anything that talks the protocol will work for us; that might change as well and then we’ll certainly start forming opinions.
Sorry to write to a topic that is more than two years old but I am pretty new to Elixir and wants to write an application that uses Kafka. I have found kafka_ex and kaffe, but do not know wich library to choose right now…
Kafka_ex seems to be changing to a new architecture … and Kaffe, well I have tried to setup a simple consumer example following the instructions with no success ( probably my fault ) , and there are no much examples out there.
Any recommendations regarding a Kafka Library in the 2020?
Thanks in advance.
I checked kafka_ex again… and KafkaEx 1.0 API seems good to me - but haven’t used kafka_ex.
BTW broadway_kafka uses brod as kafka library.
BroadwayKafka can subscribe as a consumer to one or more topics and process streams of records within the same consumer group. Communication is done through Kafka’s Consumer API using the :brod client.