Supercharging Kafka Processing with Broadway: Achieving Unmatched Parallelism and Efficiency
Comments welcome! View the elixirconf tag for more ElixirConf talks!
4 Likes
At 25:10 he says they ran into an issue of missing acknowledgments (ACKs) if they
“didn’t have enough of the producers at the beginning, to match the number of partitions, sometimes the ACKs weren’t going back. […] We wanted at least as many consumers as what we have partitions”.
(I assume that by “consumers” he meant a BroadwayKafka.Producer - it’s also shown on the slide.)
Do I understand correctly that the presenter suggest producer concurrency should be set to (at least) <sum of partitions across topics> (C for short)? But, if my reading of the source code is correct, won’t this C result in way too many group coordinators, started via :brod_group_coordinator.start_link?
Hope it’s ok to add a missing tag: broadway_kafka
