Broadway kafka - failed to join group - reason: {:nxdomain

Questions / Help
1

Hi,

I am getting the following error while connecting to kafka running in docker and port forwarded to localhost. I can shell into the docker and get topics etc. I’ve tested localhost:9092 using kafkacat and am able to list the topics etc.

00:52:17.966 [info]  Group member (some_group,coor=#PID<0.261.0>,cb=#PID<0.256.0>,generation=0):
failed to join group
reason: {:nxdomain,
 [
   {:kpro_connection, :connect, 4,
    [
      file: '/home/bw/deps/kafka_protocol/src/kpro_connection.erl',
      line: 208
    ]},
   {:kpro_connection, :init, 4,
    [
      file: '/home/bw/deps/kafka_protocol/src/kpro_connection.erl',
      line: 170
    ]},
   {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
 ]}

00:52:17.967 [info]  Group member (some_group,coor=#PID<0.261.0>,cb=#PID<0.256.0>,generation=0):
Leaving group, reason: :shutdown

Config is

producer: [                               
  module: {BroadwayKafka.Producer, [      
    hosts: [localhost: 9092],             
    group_id: "some_group",               
    topics: ["some_topic"],                    
    partition: 0,                         
  ]},                                     
  concurrency: 1                          
],                                        
processors: [                             
  default: [                              
    concurrency: 10                       
  ]                                       
]
2

nxdomain is a DNS error meaning the host name you are trying to connect to can not be resolved.

In your case it is not localhost, but rather a host provided by Kafka to your client after establishing the initial connection.

When starting the Kafka docker container, did you specify the value for the environment variable KAFKA_ADVERTISED_LISTENERS? In case yes, what was it?

You can also check what host Kafka redirects you to by running kafkacat with the -L flag:
kafkacat -b localhost:9092 -L

More details here:

1 Like
3

Thanks for your inputs. This is what I get …

$> kafkacat -b localhost:9092 -t some_topic -L
Metadata for some_topic (from broker -1: localhost:9092/bootstrap):
 1 brokers:
  broker 1001 at some-kafka:9092
 1 topics:
  topic "some_topic" with 1 partitions:
    partition 0, leader 1001, replicas: 1001, isrs: 1001

and env var
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://some-kafka:9092

4

Alright, so it looks like some-kafka is the host you can not reach.

Your actual set up might differ depending on what you need, but I can offer you one possible configuration - it’s two env variables:

        KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
        KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092

In this way kafka should be accessible via port 29092 from the inside of the docker network, and via port 9092 from your computer.