How to create Apache Kafka consumer groups?

Questions / Help
1

I’m quite new to Kafka and trying to implement a simple queue system using 1 topic and 1 consumer group assigned to that topic. However I couldn’t figure out how to create a consumer group. I’m using Kafka_ex library and Kafka version 2.4.1

2

Hey Kgllev. Why don’t you try to use Rabbitmq… it uses Erlang behind the scenes and it’s simpler than Kafka. I can show you an example if you need help because I have been using it on my projects.

I use Docker to separate things… so my rabbimq is isolated in its own container.

https://www.rabbitmq.com/tutorials/tutorial-one-elixir.html

1 Like
3

I believe this is done fully by Application config, at least it is in apps at work. The consumer_group and auto_offset_reset go into the consumer group setup, and will automatically be created if it doesn’t exist.

We recently switched to brod and have had better luck with it in production.

2 Likes
4

Thank you for the feedback :nerd_face:. Turns out the default consumer group is “kafka_ex”. I’ll take a look at brod

5

To create the consumer groups I would advise you to switch from Kafka to Cassandra. This is much easier there than with Kafka! Also, Cassandra is much bigger than Kafka, which could have a positive effect on your consumer groups!

If you don’t want to change the database completely, there is also the possibility to connect the two databases via a provider like Aiven. Aiven supports both Kafka and Cassandra, so this should not be a problem. Also, the data processing by Aiven is really fast. Could be something for you, right?

1 Like