francesco.pessina

francesco.pessina

Kafka Ex :timeout error on connector

Hello everyone :slight_smile:
I have a strange problem with Kafka Ex (GitHub - kafkaex/kafka_ex: Kafka client library for Elixir · GitHub). While running locally and connecting to a local Kafka instance, all is fine.
When deploying the application on a Kubernetes AWS cluster, I experience a weird error. The Kafka broker is an AWS managed one, and I’m sure it is reachable because other microservices (Java) can connect.

This is the error I’m experiencing:

07:05:21.853 [error] GenServer #PID<0.4672.0> terminating                                                                                                                                                │
│ ** (MatchError) no match of right hand side value: [async_commit: nil]                                                                                                                                   │
│     (kafka_ex 0.12.1) lib/kafka_ex/gen_consumer.ex:774: KafkaEx.GenConsumer.handle_message_set/2                                                                                                         │
│     (kafka_ex 0.12.1) lib/kafka_ex/gen_consumer.ex:668: KafkaEx.GenConsumer.handle_info/2                                                                                                                │
│     (stdlib 3.12.1) gen_server.erl:637: :gen_server.try_dispatch/4                                                                                                                                       │
│     (stdlib 3.12.1) gen_server.erl:711: :gen_server.handle_msg/6                                                                                                                                         │
│     (stdlib 3.12.1) proc_lib.erl:249: :proc_lib.init_p_do_apply/3                                                                                                                                        │
│ Last message: :timeout 

and this is my broker and consumer group configuration:

# Kafka brokers configuration
config :kafka_ex,
  brokers: "csv list of the brokers"

config :my_app, :children, [
  %{
    id: "consumer",
    start:
      {KafkaEx.ConsumerGroup, :start_link,
       [
         MyApp.Kafka.Consumer,
         System.get_env("KAFKA_CONSUMER_GROUP"),
         [<topic1>,<topic2>],
         [
           # setting for the ConsumerGroup
           heartbeat_interval: 1_000,
           # this setting will be forwarded to the GenConsumer
           commit_interval: 1_000,
           auto_offset_reset: :latest
         ]
       ]}
  }
]

The application is running in a Docker image built from elixir.1.11-alpine with OTP 23.

I don’t understand what’s happening :frowning:

First Post!

francesco.pessina

francesco.pessina

Nevermind, the problem may be some conflicting process… Rollbacking to a previous commit solved the issue :slight_smile:

Where Next?

Trending in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" =&gt; #BSON.ObjectId&lt;58eb1a7a9ad169198c3dXXXX&gt;, "email" =&gt; ...
New
tj0
I’ve been following the steps here for the upgrade from 1.6 to 1.7 and it has gone relatively smoothly all the way till the phoenix_view ...
New
cgraham
Hi! What is currently the best library/method for parsing text and tabular data out of PDF files in Elixir or Erlang?
New
stefanchrobot
Hi, I need a way to handle data migrations in my application. I found an article by @wojtekmach about manual migrations: Automatic and ma...
New
stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
webofbits
Squid Mesh is an open source workflow automation runtime for Elixir applications. It is aimed at Phoenix and OTP apps that want to defin...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
kip
In 2021 I started a new library called Tempo with the objective of modelling time as a set of intervals - not as instants. In 2022 I gave...
New

We're in Beta

About us Mission Statement