hauleth

hauleth

Call for testers: new Elixir Logger backend

As some of you are aware (and the rest will be if they read this post) the OTP 21 introduced new logger module which provides real-life logger to the Erlang itself. For a long time (I think that Elixir 1.7 was the first one with integration) Elixir was hooking into it to listen for messages sent by Erlang libraries, but it was only one way integration (so messages sent by Elixir wasn’t available to Erlang handlers).

But it is no more, as PR 9333 was merged to the master, now Logger is a thin wrapper over Erlang’s built in functionality with compatibility layer on top to support all backends that already are there.

However that one is a huge change and I with @josevalim were working on it for over a month to find all problems and provide good implementation. We are pretty confident with the implementation, but nothing is better than testing in production. So this is time for all of You, embrace your fears and boldly go where no one has gone before. We need You to test it with your applications (of course locally, we do not encourage anyone to test it in real production).

Please, test it, break it, report it.


For those who are interested in “what it will give us anyway?”:

  • Unified metadata between loggers (and potentially other libraries like opentelemetry)
  • Out of the box integration with Lager as Lager hooks into logger as well
  • Built in support for logging to disk via logger_disk_log_h

These are for now, in future the changes will include:

  • Additional log levels (notice, critical, alert, emergency)
  • Support for structured logging
  • Module and application level control about logs verbosity

But anyway, play with it, test it, and if you break it, let us know so we can polish it before final release of 1.10 (oh, by the way this will make Elixir 1.10 OTP 21+ only).

First Post!

josevalim

josevalim

Creator of Elixir

Most Liked

hauleth

hauleth

Yeah, it seems like there is first bug :wink: Thanks

The point is that :warn used by Elixir Logger is now :warning form the Erlang, but there should be translation from old format to the new format. This was done, but in the meantime we have changed code a little to provide given message, and the translation perished. Will provide fix in a moment.

sneako

sneako

Thanks for these contributions! I installed elixir master using asdf:

asdf install elixir ref:67dec4683de49e6e8153f9c49d0a0a179c550f9f

and gave it a try in a phoenix app where I have:

config :logger, level: :warn

And I get:

Erlang/OTP 22 [erts-10.4.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

warning: the level for Erlang's logger was set to :debug, but Elixir's logger was set to :warn. Elixir's logger value will take higher precedence
  (logger 1.10.0-dev) lib/logger/app.ex:96: Logger.App.add_elixir_handler/2
  (logger 1.10.0-dev) lib/logger/app.ex:24: Logger.App.start/2
  (kernel 6.4.1) application_master.erl:277: :application_master.start_it_old/4

** (MatchError) no match of right hand side value: {:error, {:bad_return, {{Logger.App, :start, [:normal, []]}, {:EXIT, {{:badmatch, {:error, {:invalid_level, :warn}}}, [{Logger.App, :add_elixir_handler, 2, [file: 'lib/logger/app.ex', line: 103]}, {Logger.App, :start, 2, [file: 'lib/logger/app.ex', line: 24]}, {:application_master, :start_it_old, 4, [file: 'application_master.erl', line: 277]}]}}}}}
    (mix 1.10.0-dev) lib/mix/cli.ex:106: Mix.CLI.restart_logger/0
    (mix 1.10.0-dev) lib/mix/cli.ex:82: Mix.CLI.run_task/2
    (elixir 1.10.0-dev) lib/code.ex:912: Code.require_file/2

So the first thing to note is that the log level :warn is deprecated, but is there also a new way to configure the log level?

Trying level: :info or level: :warning produce a similar warning, but no match error, and logging seems to work as usual.

dom

dom

Very excited about this. Thanks for all your work!

I’m curious what approach you have in mind for structured logging. So far I’ve been doing it this way:

  • At the call site, pass structured data via metadata:
      Logger.log(level, "Processing request",
        method: conn.method,
        path: conn.request_path,
        ...
      )
    
  • Define a custom formatter which takes all metadata:
    config :logger, :file_log,
      path: "app.log",
      level: :info,
      format: {MyApp.Logger, :format},
      metadata: :all
    
  • In the formatter, generate JSON, roughly like:
    def format(level, message, timestamp, metadata) do
     %{
        "timestamp" => format_timestamp(timestamp),
        "level" => level,
        "message" => IO.iodata_to_binary(message)
      }
      |> add_metadata(metadata)
      |> Jason.encode!()
    end
    
    where add_metadata reads each key/value pair from metadata, converts the value to a printable representation, and adds it to the event map, similar to Honeybadger.JSON.

It works, but it’s not ideal - we had to reimplement functionality from the default formatter, for instance formatting time or converting PIDs to strings. OTP reports also can’t be structured, because they’re a string blob by the time they reach the formatter (not a huge deal though).

Could we achieve JSON logging more easily with the new backend, or with future development?

Last Post!

hauleth

hauleth

You can use 'test.log' instead.

Where Next?

Popular in Discussions Top

restack_oslo
Hello, Please pardon me for any faux paux. I am 46 and this is my first time on a forum of any kind. I wanted to to get answers from tho...
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19814 150
New
nburkley
AWS re:Invent is on at the moment with some interesting announcements. One new feature in particular is the Lambda Runtime API for AWS La...
New
Fl4m3Ph03n1x
Background This question comes mainly from my ignorance. Today is Black Friday, one of my favorite days of the year to buy books. One boo...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
cvkmohan
The upcoming Phoenix 1.6 release looks very interesting. Became a habit to watch the commits - and - what they are bringing in. phx.gen...
New
AstonJ
Can you believe the first professionally published Elixir book was published just 8 years ago? Since then I think we’ve seen more books f...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

We're in Beta

About us Mission Statement