What could cause `{removed_failing_handler,'Elixir.Logger'}`

Questions / Help
,
1

In production we get {removed_failing_handler,'Elixir.Logger'} across multiple Elixir projects. I don’t get a stacktrace or anymore information than that. After that message we don’t see anymore logs until the app is redeployed. I’ve tried turning up our logger level to :debug because it looks like the erlang logger only includes more information at that level. Despite increasing the log level I still don’t see any more information or stacktrace. Any ideas what might cause this or how to get more information?

We are on elixir 1.14.1-otp-25 and erlang 25.3

We are also using LoggerJSON v5.1.2 with a formatter of LoggerJSON.Formatters.DatadogLogger

1 Like
2

The cause of this is fairly straightforward: if a log handler crashes, the system will stop sending events to it to try to stabilize things.

What kind of metadata / context are you setting in the logger config? I could see this happening if the formatter tries to encode a term to JSON that doesn’t support it…

1 Like
3

Thanks for looking at this.

My prod logger config is like this:

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]

config :logger_json, :backend,
  metadata: :all,
  json_encoder: Jason,
  formatter: LoggerJSON.Formatters.DatadogLogger

config :logger,
    backends: [LoggerJSON],
    level: :info,
    handle_sasl_reports: true

The JSON payload includes span_id, trace_Id, domain, erl_level, logger filename, logger line, logger method_name, logger thread_name, message, request_id, syslog hostname, syslog severity, syslog timestamp