alexandru-calinoiu
I have a production app that uses commanded, the event handlers are started using a supervisor with the default retry values (max restarts 3, max seconds 5).
The problem is that if one of the command event handlers restarts more than 3 times in 5 seconds it will take the entire app down with it because it will kill all the other event handlers and then all the way up to the web server.
What are my options to solve this issue?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
LostKobrakai
Unrelated to commanded specifically, but using the supervision tree to handle expected failures is not a great idea. The restart behaviour of supervisors is meant to keep a system up and running (as in available) and if it fails to do so stop trying, so someone higher up can take a look. Eventually that’s someone on call. That’s what you’re seeing. You could allow more failures to happen before giving up, but at least after a few retries not having backoff is considered waistful.
Generally you should figure out what you do want to happen instead of letting the app fail and the tradeoffs involved. With that you can start looking into how to best implement that, especially given commanded is likely the tool to queue up the events for processing. Not sure what options it provides to deal with failing events over time.
I’d also suggest considering what classes of errors might result in errors of your event handlers. E.g. a bug is unlikely to resolve without a software update, a network issue might resolve over time. You might even want to handle them differently per type of error.
slashdotdash
You should implement the
error/3callback function in event handlers to handle problematic events so that the event handler does not restart on error.The example error handling in the docs shows a simple strategy to retry X times and then log and skip an event that cannot be handled.
xpg
We ended up solving this by setting up a supervisor that allows our event handlers to crash, without restarting them. At runtime, we can query the supervisor about which event handlers / projectors are stopped and not restarted.
Our operations team is notified of this, so that they can investigate the situation.
The reason we did not go with skipping events, is that we’re afraid of continuing handling events, if there is one we cannot handle.
The only time we have needed this, it worked like a charm: We had an event being introduced by accident, that was unhandled in a projector. The projector got stuck and died. The rest of the system kept running, but operations were aware of the issue. We could then handle the situation (which in this case was done by blanking out the event), and restart the projector.