darnahsan

darnahsan

Broadway Kafka only one 1 node consuming messages

I have a phoenix app running in 2 containers both connected to redpanda(zookeeper less kafka) but I only see 1 node consuming messages from the topic, topic has only 1 partition. Below is my setup code. If I understand correctly both should consume messages as they are part of the same (consumer) group

@redpanda_port 9092
  @redpanda_host "redpanda"
  @redpanda_nodes [{@redpanda_host, @redpanda_port}]
  @redpanda_group_id "stream_00"
  @redpanda_topic "stream"
  @producer_concurrency 1
  @processors_concurrency 10

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {BroadwayKafka.Producer,
           [
             hosts: @redpanda_nodes,
             group_id: @redpanda_group_id,
             topics: [@redpanda_topic],
             client_id_prefix: nodename()
           ]},
        concurrency: @producer_concurrency
      ],
      processors: [
        default: [
          concurrency: @processors_concurrency
        ]
      ]
    )
  end

Marked As Solved

darnahsan

darnahsan

seems its a kafka limitation when there is 1 partition only 1 consumer is assigned

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New

We're in Beta

About us Mission Statement