kgllev
How to create Apache Kafka consumer groups?
I’m quite new to Kafka and trying to implement a simple queue system using 1 topic and 1 consumer group assigned to that topic. However I couldn’t figure out how to create a consumer group. I’m using Kafka_ex library and Kafka version 2.4.1
Marked As Solved
sb8244
I believe this is done fully by Application config, at least it is in apps at work. The consumer_group and auto_offset_reset go into the consumer group setup, and will automatically be created if it doesn’t exist.
We recently switched to brod and have had better luck with it in production.
Also Liked
tiagodavi
Hey Kgllev. Why don’t you try to use Rabbitmq.. it uses Erlang behind the scenes and it’s simpler than Kafka. I can show you an example if you need help because I have been using it on my projects.
I use Docker to separate things.. so my rabbimq is isolated in its own container.
Big_chris
To create the consumer groups I would advise you to switch from Kafka to Cassandra. This is much easier there than with Kafka! Also, Cassandra is much bigger than Kafka, which could have a positive effect on your consumer groups!
If you don’t want to change the database completely, there is also the possibility to connect the two databases via a provider like Aiven. Aiven supports both Kafka and Cassandra, so this should not be a problem. Also, the data processing by Aiven is really fast. Could be something for you, right?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"