axelson

axelson

Scenic Core Team

How to debug telemetry handlers that have errored

Is there a way to get Telemetry to output the full exception when a handler crashes?

It’s a little difficult to debug when you only get a message like this:

[error] Handler "my-app-handler-id" has failed and has been detached. Class=:error
Reason=:undef
Stacktrace=[
  {Logger, :info, ["GET /posts served in 2ms"], []},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [
  ...

In particular the Reason=:undef feels cryptic compared to a normally formatted exception, but also it would be easier to read the stacktrace if it was formatted as if from Exception.format_stacktrace/1.

Also is there an easy way to re-attach the handler so I can test the updated code without restarting the server?

Marked As Solved

D4no0

D4no0

I don’t know if that is idiomatic, however the way I used to debug this back in the day was to wrap the handler code in a try/catch block and just print the error in the console.

Also Liked

LostKobrakai

LostKobrakai

That’s erlang errors in a nutshell – it’s the error for module or function undefined. We usually don’t see them because of the elixir logger translator if this bubbles up, but telemetry doesn’t let this bubble to not break the place the handler was called in. And with it being an erlang library it also cannot use any of the elixir level translation after having cought the error.

What might be possible is having an telemetry_translator elixir library, which can translate the telemetry logs specifically by extracting the relevant data.

axelson

axelson

Scenic Core Team

Ah, that’s a great tip! I forget how I introduced the original error in the OP but by adding this to the end my telemetry handler:

  rescue
    e ->
      Logger.error(Exception.format(:error, e, __STACKTRACE__))
  end

I can turn this error:

[error] Handler "my-app-handler-id" has failed and has been detached. Class=:error
Reason=%Protocol.UndefinedError{protocol: Enumerable, value: nil, description: ""}
Stacktrace=[
  {Enumerable, :impl_for!, 1, [file: ~c"lib/enum.ex", line: 1]},
  {Enumerable, :reduce, 3, [file: ~c"lib/enum.ex", line: 166]},
  {Enum, :map, 2, [file: ~c"lib/enum.ex", line: 4423]},
  {MyAppWeb.Telemetry, :handle_event, 4,
   [file: ~c"lib/my_app_web/telemetry.ex", line: 127]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [
     file: ~c"/Users/jason/dev/my_app/deps/telemetry/src/telemetry.erl",
     line: 167
   ]},
  {:lists, :foreach_1, 2, [file: ~c"lists.erl", line: 1686]},
  {Plug.Telemetry, :"-call/2-fun-0-", 4,
   [file: ~c"lib/plug/telemetry.ex", line: 76]},
  {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]},
  {Plug.Conn, :run_before_send, 2, [file: ~c"lib/plug/conn.ex", line: 1850]},
  {Plug.Conn, :send_resp, 1, [file: ~c"lib/plug/conn.ex", line: 441]},
  {MyAppWeb.PostController, :action, 2,
   [file: ~c"lib/my_app_web/controllers/post_controller.ex", line: 1]},
  {MyAppWeb.PostController, :phoenix_controller_pipeline, 2,
   [file: ~c"lib/my_app_web/controllers/post_controller.ex", line: 1]},
  {Phoenix.Router, :__call__, 5, [file: ~c"lib/phoenix/router.ex", line: 484]},
  {MyAppWeb.Endpoint, :plug_builder_call, 2,
   [file: ~c"lib/my_app_web/endpoint.ex", line: 1]},
  {MyAppWeb.Endpoint, :"call (overridable 3)", 2,
   [file: ~c"/Users/jason/dev/my_app/deps/plug/lib/plug/debugger.ex", line: 136]},
  {MyAppWeb.Endpoint, :call, 2, [file: ~c"lib/my_app_web/endpoint.ex", line: 1]},
  {Phoenix.Endpoint.SyncCodeReloadPlug, :do_call, 4,
   [file: ~c"lib/phoenix/endpoint/sync_code_reload_plug.ex", line: 22]},
  {Bandit.Pipeline, :call_plug!, 2,
   [file: ~c"lib/bandit/pipeline.ex", line: 124]},
  {Bandit.Pipeline, :run, 4, [file: ~c"lib/bandit/pipeline.ex", line: 36]},
  {Bandit.HTTP1.Handler, :handle_data, 3,
   [file: ~c"lib/bandit/http1/handler.ex", line: 12]}
]

Into this much easier to read error:

[error] ** (Protocol.UndefinedError) protocol Enumerable not implemented for nil of type Atom. This protocol is implemented for the following type(s): DBConnection.PrepareStream, DBConnection.Stream, Date.Range, Ecto.Adapters.SQL.Stream, File.Stream, Function, GenEvent.Stream, HashDict, HashSet, IO.Stream, Jason.OrderedObject, List, Map, MapSet, Phoenix.LiveView.LiveStream, Postgrex.Stream, Range, Stream
    (elixir 1.17.2) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir 1.17.2) lib/enum.ex:166: Enumerable.reduce/3
    (elixir 1.17.2) lib/enum.ex:4423: Enum.map/2
    (my_app 0.1.0) lib/my_app_web/telemetry.ex:127: MyAppWeb.Telemetry.handle_event/4
    (telemetry 1.3.0) /Users/jason/dev/my_app/deps/telemetry/src/telemetry.erl:167: anonymous fn/4 in :telemetry.execute/3
    (stdlib 5.2.3.1) lists.erl:1686: :lists.foreach_1/2
    (plug 1.16.1) lib/plug/telemetry.ex:76: anonymous fn/4 in Plug.Telemetry.call/2
    (elixir 1.17.2) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
    (plug 1.16.1) lib/plug/conn.ex:1850: Plug.Conn.run_before_send/2
    (plug 1.16.1) lib/plug/conn.ex:441: Plug.Conn.send_resp/1
    (my_app 0.1.0) lib/my_app_web/controllers/post_controller.ex:1: MyAppWeb.PostController.action/2
    (my_app 0.1.0) lib/my_app_web/controllers/post_controller.ex:1: MyAppWeb.PostController.phoenix_controller_pipeline/2
    (phoenix 1.7.14) lib/phoenix/router.ex:484: Phoenix.Router.__call__/5
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
    (my_app 0.1.0) /Users/jason/dev/my_app/deps/plug/lib/plug/debugger.ex:136: MyAppWeb.Endpoint."call (overridable 3)"/2
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
    (phoenix 1.7.14) lib/phoenix/endpoint/sync_code_reload_plug.ex:22: Phoenix.Endpoint.SyncCodeReloadPlug.do_call/4
    (bandit 1.5.7) lib/bandit/pipeline.ex:124: Bandit.Pipeline.call_plug!/2
    (bandit 1.5.7) lib/bandit/pipeline.ex:36: Bandit.Pipeline.run/4
    (bandit 1.5.7) lib/bandit/http1/handler.ex:12: Bandit.HTTP1.Handler.handle_data/3

And I think with the original Reason=:undef error the difference would be even more drastic!

I’ve marked your post as the solution since it’s a very practical way to approach this, but I’d still love an easier way to debug this without having to modify the handler itself.

Now I just need to remember to not accidentally commit that change :laughing:

LostKobrakai

LostKobrakai

What is a “full error message” though. What you’re seeing is afaik all you get for erlang errors. It’s also all the elixir logger translator would see and turn into a more friendly sounding exception, but still without more details.

Elixir however also has the Exception.blame callback, which for exceptions can be used to include more context, which might be too expensive to produce in production environments. That’s again a pure elixir concept though.

That’s a great tip, especially given I doubt telemetry would add an option for customizing this.

With reraise you even get the same behaviour against telemetry instead of hiding the failure.

Where Next?

Popular in Questions Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New

Other popular topics Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 48342 226
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement