Configuring Logger Handlers at Runtime

Questions / Help
, , ,
1

I am trying to use the third-party schlagert/syslog erlang library for sending logs to Papertrail and running into some inconsistencies and issues around configuration.

If I configure it as follows, it sends logs to Papertrail correctly:

config :syslog,
  protocol: {:rfc5424, :udp},
  dest_host: ~c"my.papertrail.url,
  dest_port: my_papertrail_port,
  facility: :local0,
  app_name: "my_app_name",
  formatter: Logger.Formatter.new(
    format: "$message $metadata\n", metadata: [...]
  )

But, if I follow the documented instructions for integrating Erlang/OTP Handlers and use the exact same configuration I am able to see the correct handler and configuration when running iex> :logger.get_handler_config(), but no logs are sent to Papertrail.

config :my_app, :logger, [
  {:handler, :papertrail, :syslog_logger_h,
   %{
     config: %{
       protocol: {:rfc5424, :udp},
       dest_host: ~c"my.papertrail.url",
       dest_port: my_papertrail_port,
       facility: :local0,
       app_name: "my_app"
     },
     formatter:
          Logger.Formatter.new(
            format: "$message $metadata\n",
            metadata: [:event, :result, :context]
          )
   }}
]

Note, when I do it this recommended way, I do call Logger.add_handlers(:my_app) from my Application.start/2 function. Also, when I do this I have two handlers from syslog, one called :syslog and the other called :papertrail. If I try to use the above code but replace :papertrail with :syslog it fails with an error saying that handler was already added.

I wouldn’t mind doing things the way that works, but the issue is that I have to hard-code the values for dest_host and dest_port in my config.exs module. If I omit those (or set them to dummy values) and then call :logger.update_handler_config in my Application.start/2 function it shows the updated handler config but doesn’t actually seem to affect what :syslog does as I don’t see any logs in Papertrail. So, there doesn’t seem to be any good way to configure this custom handler at runtime. I’ve tried calling :logger.update_handler_config for both :syslog and :papertrail and neither works.

One hypothesis I have is that :syslog is started by Mix because it’s in my deps function. When starting, it looks into my config.exs file for values and initializes with these values. It’s outside of my application in a way so calling :logger.update_handler_config doesn’t work. But, another oddity is that if I call :logger.update_handler_config and just change the formatter: configuration, that does take effect. So, it seems that I can’t update some configuration values while others do work.

I’m at a loss. All I want to do is send logs to Papertrail and it’s turning out to be far more complex than anticipated.

2 Likes
2

I ran some tests and then reviewed the source code. It appears that it is not possible to specify dest_host and dest_port per handler.

I’m using this listener as my syslog:

socat -u UDP-RECV:12345,fork SYSTEM:'tee -a /tmp/syslog.txt'

Logging in iex with the following configuration sends two logs to my syslog.txt:

# Disable the default logger
config :syslog, [{:logger, []}]

# It seems only this dest_port is read from the config
config :syslog, dest_port: 12345

config :syslog_test, :logger, [
  {:handler, :one, :syslog_logger_h, %{
     config: %{
       # This is ignored
       dest_port: 123456
     }
   }},
  {:handler, :two, :syslog_logger_h, %{
     config: %{
       # This is ignored
       dest_port: 123456
     }
   }}
]

iex> require Logger
iex> Logger.add_handlers(:syslog_test)
iex> Logger.info("hello")

When I run Logger.info("hello"), two messages are sent via UDP to my listener on port 12345, appending two hello logs to my file.

Looking at the logger handler code in the library here:

Following the logs, you’ll see that the syslog_logger is a gen_server responsible for sending the log from all handlers over UDP.

This gen_server is launched at startup, and its state contains the dest_port and dest_host, which cannot be changed at runtime.

The library would need to be modified to support different dest_host and dest_port per handler.