Good day everyone. We are using Broadway/Kafka.
Most of the time, our consumers handle messages under 20ms, but sometimes we see spikes in latency up to 700-800ms. The weird thing is that as per broadway.processor.message.stop.duration we always handle messages under 20 ms. It is super important for us to have as minimum latency as possible.
We measure latency as a timestamp which is set by other Kafka producers on a message sending minus inserted_at of a resulting struct.
I’m sure that the problem somewhere inside Broadway or Kafka configuration, since our observations regarding latency are confirmed by manual or load tests.
Q: What do you think where is the problem could be?
Amount of partitions: 15
Configuration:
Broadway.start_link(__MODULE__,
name: __MODULE__,
producer: [
module: {
Kafka.config!(:broadway_producer),
[
hosts: Kafka.config!(:hosts),
group_id: Kafka.config!(:group_id),
topics: @topics,
receive_interval: Kafka.config!(:receive_interval), # 0ms
]
},
concurrency: Kafka.config!(:producer_concurrency) # 15ms
],
processors: [
default: [concurrency: Kafka.config!(:processor_concurrency)] #10 ms
]
)
Any ideas?
