chulkilee

MetadataLogger: logging with metadata (+ tesla / plug middleware)

Hi all,

Here are several libraries I wrote to do structured logging with logger metadta.

metadata_logger
metadata_logger_plug: middleware for plug
metadata_logger_tesla: middleware for tesla

(There were in different name - now renamed to metadata_logger_* and moved to own github org)

Highlights

Formatter, not backend; e.g. it doesn’t reimplement console logger backend. Choose your own backend! (or just console)
Minimal transformation: if you pass struct into metadata, it won’t convert it to map blindl; this is to prevent from leaking data into logs by accident. It uses Jason
Return IO data (not building a large binary!)

Most Liked

hauleth

As you correctly pointed to my post on the forum with call for testers I need to make few statements:

In near future (when the new logger implementation stabilises) we will add proper structured logging to the Elixir’s Logger where metadata and “log message” (or, as called by Erlang, report) will be possible without any wrappers (#9465)
There is plan to deprecate “old” backends (and naturally formatters) in favour of using Erlang’s handlers and formatters API directly (#7928)

Additionally your metadata_logger_plug is currently more or less possible via Plug.Telemetry API, which I believe is currently more standardised and commonly supported (ex. Phoenix.Logger uses that).

However, nice work and I really appreciate idea of bringing structured logging to Elixir.

Post #2

chulkilee

Thanks! I know there are lots of work going on logging and telemetry (which can be used for logging as well).

Here are my takes:

Log formatter will remain - just in different form: seems like new erlang logger (21.1) has log_formatter and Elixir is planning to use same mechanism so that elixir and erlang use same log event, and leverage same logging backend (via passing handler_config)

Metadata will be a map - I think this is the right way to go, as erlang already uses map ref (and it’s planned in #7928)

Passing map as a message is planned but it won’t invalidate use of metadata - passing a map as a message is planned in #9465. However, I see metadata is for scope-based while map message is for per message - and I found both are needed when I’m using them.

Note that this can be done even in the current Elixir if you use formatter taking a map message See this PR I just wrote…

instrumentation logging via :telemetry - I really like that idea! Plug.Telemetry was added after I wrote this middleware I created issues: for plug and for tesla

Post #3

hauleth

I am fully aware of the implementation and future plans, I made both of them

Yes, but the plan is to make it Erlang logger formatter, as you have noticed.

Internally, yes, from the user viewpoint, no. This is required for the compatibility. So in Elixir 1.10 it will be changed before returning in Logger.metadata/0.

Yes, as metadata is for something completely different. Using your example in metadata_logger_plug the status, body length, and response status should be part of the report, not metadata. Metadata is more about, well, meta data like: version, log position, module, function, application name, etc.

Well, partially. It will work for that particular formatter, but in general it will fail (also Dialyzer will complain) as you break contract about type of the accepted data. So that single formatter will work (because it relies on undefined behaviour), but any other will fail, and this will fail with parsing Erlang reports, as these are handled before these end in your translator.

Post #4