Broadway Kafka only one 1 node consuming messages

Questions / Help
, ,
1

I have a phoenix app running in 2 containers both connected to redpanda(zookeeper less kafka) but I only see 1 node consuming messages from the topic, topic has only 1 partition. Below is my setup code. If I understand correctly both should consume messages as they are part of the same (consumer) group

@redpanda_port 9092
  @redpanda_host "redpanda"
  @redpanda_nodes [{@redpanda_host, @redpanda_port}]
  @redpanda_group_id "stream_00"
  @redpanda_topic "stream"
  @producer_concurrency 1
  @processors_concurrency 10

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {BroadwayKafka.Producer,
           [
             hosts: @redpanda_nodes,
             group_id: @redpanda_group_id,
             topics: [@redpanda_topic],
             client_id_prefix: nodename()
           ]},
        concurrency: @producer_concurrency
      ],
      processors: [
        default: [
          concurrency: @processors_concurrency
        ]
      ]
    )
  end
1 Like
2

seems its a kafka limitation when there is 1 partition only 1 consumer is assigned

1 Like