New Relic Plug handler fails to process certain telemetry events due to a :badkey error

Questions / Help
, ,
1

The problem

New Relic Plug handler fails to process certain telemetry events due to a :badkey error. This results in the handler being detached and further telemetry collection disrupted. I am using Bandit v1.5

Logs

Log: [error] Handler {:new_relic, :plug} has failed and has been detached. Class=:error
Reason: {:badkey, :conn,
%{
  telemetry_span_context: #Reference<0.1763085910.242745346.37981>,
  connection_telemetry_span_context: #Reference<0.1763085910.242745346.7102>
}}

Stacktrace:
[
  {NewRelic.Telemetry.Plug, :get_headers, 2,
   [file: "lib/new_relic/telemetry/plug.ex", line: 272]},
  {NewRelic.Telemetry.Plug, :handle_event, 4,
   [file: "lib/new_relic/telemetry/plug.ex", line: 102]},
  {:telemetry, :"execute/3-fun-0", 4,
   [file: "/app/deps/telemetry/src/telemetry.erl", line: 167]},
  {:lists, :foreach_1, 2, [file: "lists.erl", line: 1686]},
  {Bandit.Telemetry, :start_span, 3, [file: "lib/bandit/telemetry.ex", line: 168]},
  {Bandit.Pipeline, :run, 4, [file: "lib/bandit/pipeline.ex", line: 53]},
  {Bandit.HTTP1.Handler, :handle_data, 3,
   [file: "lib/bandit/http1/handler.ex", line: 12]},
  {Bandit.DelegatingHandler, :handle_data, 3,
   [file: "lib/bandit/delegating_handler.ex", line: 18]}
]

Environment

  • (elixir: "~> 1.14",)
  • {:phoenix, "~> 1.7.14"}
  • {:new_relic_agent, "~> 1.30.0"}
  • {:new_relic_oban, "~> 0.0.2"}
  • {:bandit, "~> 1.5"}

Is there to gracefully handle that scenario ? Is there more logging that we need to figure this out ?

Any help would be much appreciated :slight_smile:

2

This seems like a bug to me, you should open a issue on the project repo.

3

I did, but didn’t get a response. this is a desperation move :frowning:

4

Are you handling the telemetry events in your application code or this is some kind of under the hood thing the library does?

If yes, then you can just wrap your handle code in a try/catch block. I guess you could do the same with the library, fork it and wrap it in the same way, it will avoid at least detaching of the handler.

5

it is a under the hood the lib does, weirdly enough it happens at random times every once in a while, I will enable debug logs and try to see if there’s a specific request format that’s causing this.

If all else fails, I’ll probably go back to using Cowboy, or fork the lib

Thanks!

6

Your best would probably be to fork and then make a PR after you make it work properly.

2 Likes