I’m quite new to Kafka and trying to implement a simple queue system using 1 topic and 1 consumer group assigned to that topic. However I couldn’t figure out how to create a consumer group. I’m using Kafka_ex library and Kafka version 2.4.1
Hey Kgllev. Why don’t you try to use Rabbitmq… it uses Erlang behind the scenes and it’s simpler than Kafka. I can show you an example if you need help because I have been using it on my projects.
I use Docker to separate things… so my rabbimq is isolated in its own container.
I believe this is done fully by Application config, at least it is in apps at work. The consumer_group and auto_offset_reset go into the consumer group setup, and will automatically be created if it doesn’t exist.
We recently switched to brod and have had better luck with it in production.
Thank you for the feedback . Turns out the default consumer group is “kafka_ex”. I’ll take a look at brod
To create the consumer groups I would advise you to switch from Kafka to Cassandra. This is much easier there than with Kafka! Also, Cassandra is much bigger than Kafka, which could have a positive effect on your consumer groups!
If you don’t want to change the database completely, there is also the possibility to connect the two databases via a provider like Aiven. Aiven supports both Kafka and Cassandra, so this should not be a problem. Also, the data processing by Aiven is really fast. Could be something for you, right?