Thanks for sharing! I feel your pain! This isn’t exactly an issue with Broadway itself, but rather a conceptual mismatch between how Kafka operates and certain use cases.
These libraries are designed to work with message queue systems where messages can be individually acknowledged, sometimes even out of order
Kafka, on the other hand, is a streaming system where typical consumers stay at a particular offset and won’t advance unless messages are acknowledged in sequence.
This characteristic of Kafka forces a choice:
- Don’t acknowledge, and the flow stops.
- Acknowledge and move on, even if the message wasn’t properly handled.
If you acknowledge a message that wasn’t fully processed, Kafka doesn’t have an out-of-the-box solution for this—unlike some queue systems.
To handle this purely with Kafka, here are a couple of approaches:
-
Reproduce the acknowledged-but-unprocessed message to the topic: This isn’t always ideal, but if your system is the only one consuming from the topic and can handle duplicates, it can work.
-
Use a retry topic to store the “acknowledged but not processed” messages, then consume them in a separate flow: If the consume here fails again, you can follow the first approach on this topic, because you will be the only one consuming it.
My goal with Klife’s consumer design is to eventually include these features out of the box. Ideally, something like:
use Klife.Consumer, retry_topic: {true, retry_config}
The retries would then be managed automatically, though it may be tricky to implement. Let’s see how it goes!
