alexandru-calinoiu

alexandru-calinoiu

How to retry failed Event Handlers in Commanded

I have a production app that uses commanded, the event handlers are started using a supervisor with the default retry values (max restarts 3, max seconds 5).

The problem is that if one of the command event handlers restarts more than 3 times in 5 seconds it will take the entire app down with it because it will kill all the other event handlers and then all the way up to the web server.

What are my options to solve this issue?

Most Liked

slashdotdash

slashdotdash

You should implement the error/3 callback function in event handlers to handle problematic events so that the event handler does not restart on error.

The example error handling in the docs shows a simple strategy to retry X times and then log and skip an event that cannot be handled.

xpg

xpg

We ended up solving this by setting up a supervisor that allows our event handlers to crash, without restarting them. At runtime, we can query the supervisor about which event handlers / projectors are stopped and not restarted.
Our operations team is notified of this, so that they can investigate the situation.

The reason we did not go with skipping events, is that we’re afraid of continuing handling events, if there is one we cannot handle.

The only time we have needed this, it worked like a charm: We had an event being introduced by accident, that was unhandled in a projector. The projector got stuck and died. The rest of the system kept running, but operations were aware of the issue. We could then handle the situation (which in this case was done by blanking out the event), and restart the projector.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
hariharasudhan94
Lets say i have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => "XXX...
New

We're in Beta

About us Mission Statement