OTP GenStage/GenServer crash notification

Questions / Help
, ,
#1

I’ve created an OTP application using Supervised GenStage producer/consumer model and would like to add in the ability to get notified (via email and/or pagerduty) if any supervised processes fail.

I don’t have any exception handling code, so all exceptions and errors that I haven’t specifically handled using the {:ok, term}, {:error, term} mechanims should bubble to the top and cause a crash, which the supervisor would notice a restart the application.

I would like to know when this happens via some notification.

I originally thought about adding something in the supervisor, but I read that is frowned upon. Supervisors should be as minimal as possible.
I also read the GenStage/GenServer terminate callback doesn’t always get executed during a crash so that also is not an option.

What’s the best/canonical way to handle this?

#2

The built-in Erlang SASL application will log every-tiny-little-thing that happens in a supervision tree in regards to creation and death, it will overwhelm your logs if you log it, but you can listen to it straight to react to it, however if you have a very busy app then it may actually become the bottleneck. ^.^

The usual way to listen for that though is to have a process that ‘monitors’ whatever specific processes you are interested it, then you will get messages when they die and why. But you have to do your own book-keeping and all.

#3

I’ll go with adding a monitor to the supervision tree, thanks for the response.