stavro
In general, the docs for GenStage seem to have very minimal documentation around how to deal with errors, what to expect in the presence of errors, at most once vs at least once delivery of events, etc. I’m having a hard time understanding the guarantees that GenStage provides in the presence of errors during stages.
For example, a textbook implementation of GenStage.BroadcastDispatcher (seen here: 0_usage.md · GitHub) seems to have pretty lackluster behavior in the face of errors during a consumer stage. In a simple test, if I enqueue 10 events, and a single consumer has an error on the first one, the next 9 appear to never even be attempted.
Since the consumer is supervised independently, and only received a single event to process (as seen in the debug logs), why are the other nine events not handled by the consumer after its supervisor restarts it?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
At WyeWorks, we believe great software is built by combining technical excellence with genuine human connection
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 7- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
josevalim
That’s because it does not give any guarantees.
If there is an error, you lose everything that is in the message queue and you didn’t persist elsewhere yourself, as in any other Elixir process or abstraction, such as a GenServer.
stavro
Got it - I misunderstood and thought that the message queue was kept outside of the consumer process.
Hmm. Ok. Thanks!
mbuhot
Are there any recommendations for an at-least once delivery of messages?
Eg if you had a pipeline with:
RabbitMQ → GenStage → GenStage → GenStage → Database
And you want to ensure that every message from the RabbitMQ is processed by the GenStage pipeline and written to the database.
Seems like you would need to keep track of the original message ID through the pipeline and only send the ACK to RabbitMQ it after writing to the database.
pma
The solution I’m using is to have the delivery_tag as part of the event and acknowledging to RabbitMQ only after the last stage: wabbit/examples/direct_exchange.ex at master · pma/wabbit · GitHub
madshargreave
Did you come up with a solution for this? I am in the same situation, although with Kafka
mbuhot
Haven’t had to connect a multi-step GenStage pipeline to Kafka or Rabbit yet. But I would approach it the same way, updating the consumer offsets in the last stage of the pipeline.
Which Kafka client are use using?
madshargreave
@mbuhot Sorry for the late reply. I am using the Brod erlang library