kgllev
I’m quite new to Kafka and trying to implement a simple queue system using 1 topic and 1 consumer group assigned to that topic. However I couldn’t figure out how to create a consumer group. I’m using Kafka_ex library and Kafka version 2.4.1
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
tiagodavi
Hey Kgllev. Why don’t you try to use Rabbitmq.. it uses Erlang behind the scenes and it’s simpler than Kafka. I can show you an example if you need help because I have been using it on my projects.
I use Docker to separate things.. so my rabbimq is isolated in its own container.
https://github.com/pma/amqp
sb8244
I believe this is done fully by Application config, at least it is in apps at work. The consumer_group and auto_offset_reset go into the consumer group setup, and will automatically be created if it doesn’t exist.
We recently switched to brod and have had better luck with it in production.
kgllev
Thank you for the feedback
. Turns out the default consumer group is “kafka_ex”. I’ll take a look at brod
Big_chris
To create the consumer groups I would advise you to switch from Kafka to Cassandra. This is much easier there than with Kafka! Also, Cassandra is much bigger than Kafka, which could have a positive effect on your consumer groups!
If you don’t want to change the database completely, there is also the possibility to connect the two databases via a provider like Aiven. Aiven supports both Kafka and Cassandra, so this should not be a problem. Also, the data processing by Aiven is really fast. Could be something for you, right?