Is it possible to have different log levels for different applications/modules?

[svsdehh]

is it possible in elixir to have different log levels for different applications/modules? Or is this only a global setting?

We have a microservice that uses phoenix and would like to log some information in our application in the info level. but when setting the level to info phoenix is logging every response time on info and is therefore logging way to much.

I would like to log phoenix on warn or error level but have our own application using info level. Is this possible? With the standard logger? With an alternative implementation?

[kokolegorille]

Looking at the Logger doc…

The supported levels are:

:debug - for debug-related messages
:info - for information of any kind
:warn - for warnings
:error - for errors

Why do You need to use info only?

iex> Logger.debug "yo"
[debug] yo
:ok
iex> Logger.info "yo" 
[info] yo
:ok

[NobbZ]

He doesn’t use only info, he does use warn and error as well.

Having different severity for different sub-packages/modules/units/whatever-your-language-uses-to-organize-code seems to be a common idiom.

I’ve seen libraries dealing with such requirements in go, C, C++, rust, ruby, and many others, but not in elixir so far.

But to be honest. Logger is fine to get started, but if one has some requirements, then one should search for something else.

Eg for most of my code I do at my dayjob, I do have the requirement of 5 levels, also we log in single-line JSON.

---

FYI our loglevels are:

• critical: the logged event can massively disturb the complete network of applications. A vanishing load balancer or DNS could be such an event
• error: the logged event can massively disturb the application the event occured in. A database server not beeing able to access its files could be such an event
• warning: an error happened, but it is easily recoverable. Nearly everything you would repair by restarting the process in OTP can fit here
• info: the event is just logged as an information for the enduser, like receiving a request from the client
• debug: accompanies rough steps while processing a request.

We do not use elixir though at the moment Therefore I wasn’t in the need to search or implement alternatives to logger

[svsdehh]

we are not using info only. but phoenix is logging things on the info level, that does not interest us. but beside error and warning we are also using info level. I want to see the logging from our info level, but not the logging on the info level from e.g. phoenix. the phoenix logging an logging from other applications would just bloat the logging.

but I do not want to set the logging level to warn, just to suppress the logging from other applications, as then I would not see our own info logging any more.

I think this is a standard use case.

[svsdehh]

@NobbZ

I’ve seen libraries dealing with such requirements in go, C, C++, rust, ruby, and many others, but not in elixir so far.

I thought that this is such a standard use case, that every logger should have it. especially considering the fact, that elixir is not an old language.

As there are some companies using elixir in production, how are they logging? Can anyone make some suggestions?

[NobbZ]

Most languages don’t even have a propper logger in there stdlib. Even for elixir it is a dependency you have to opt in for explicitely.

None of the languages I’ve dealt with so far had a logger capable of your requirements in the stdlib. Most are just very simple wrappers around something that is similar to IO.puts "#{:os.system_time} #{message}" while pretty printing the system-time.

[svsdehh]

but most languages are older than elixir. as a new Logger implementation I had hoped, that it would implement what is needed for monitoring today.

And as you did not know an implementation that fulfill our requirement, it seems, that it does not exists…

[NobbZ]

I’d not go as far to say it doesn’t exist. It’s just that I haven’t searched so far since in elixir world I’m just a hobbyist and happy with what :logger provides me

What I’ve seen though in a legacy project I had to take over, the main process dumped everything as XML into a named pipe. Another Program running on the same system filtered those data and then wrote those data into files according to loglevels. But none of those tools was written in elixir, still it may be possible that a similar architecture suits you as a quick get it running and then do a propper logger.

[jlevy]

Logger is a modular system which allows you to implement your own backend to format, store, print, or send log entries however you would like. A custom backed is basically an event handler. For every log event, it receives some metadata including the module, function, and process that generated the log entry. I could see writing your own backend that ignores the Logger configured log level and has its own more granular configuration scheme. Your backend can handle what it wants to keep and drop what it doesn’t. This is discussed in the Logger docs.

It’s important to note though that the Logger configured log level would still be used at compile time to remove Logger calls below that level. In other words, Phoenix will still call every one of those Info calls and perf will be impacted accordingly. This is in contrast to if you set the Logger level to warn - in this case the Info calls are removed at compile time and don’t exist at runtime.

[svsdehh]

@jlevy

This is in contrast to if you set the Logger level to warn - in this case the Info calls are removed at compile time and don’t exist at runtime.

but only if I use compile_time_purge_level or not?

@jlevy

Your backend can handle what it wants to keep and drop what it doesn’t. This is discussed in the Logger docs.

do I get who calls the logger – so that I can then decide what to drop an what not?

[svsdehh]

thanks for your suggestions. somehow I had the hope, I could just use the logger and must not write an own one

[peerreynders]

Example implementation: onkel-dirtus/logger_file_backend as it was mentioned in

Configuring Logger on Umbrella

I think the only way you would achieve this is to use a custom backend, and then use the :application or :module metadata keys to do something with it. Backends like https://github.com/onkel-dirtus/logger_file_backend has a metadata_filter config option that you can use to filter out messages to certain files. Alternatively if you want it in a console, it shouldn’t be that hard to knock up what you need: Copied the main Logger.Backend.Console from https://github.com/elixir-lang/elixir/blob/…

[dom]

Have you tried setting the level to debug on Plug.Logger in your endpoint?

[svsdehh]

@peerreynders

Example implementation: onkel-dirtus/logger_file_backend as it was mentioned in

I hoped that this is not necessary. we are already using a different logger (gelf).

@dom

Have you tried setting the level to debug on Plug.Logger in your endpoint?

thanks for the hint. i will try to set that debug level to “warn”. this is not a general solution but maybe fix the problem with the bloat from the response time.

EDIT: i did not understand the docu in the first read. you are right, is has to be debug!

[peerreynders]

@svsdehh

I hoped that this is not necessary. we are already using a different logger (gelf).

Between

• gelf_logger/lib/gelf_logger.ex at master · jschniper/gelf_logger · GitHub and
• logger_file_backend/lib/logger_file_backend.ex at master · onkel-dirtus/logger_file_backend · GitHub

you have a pretty good head start towards creating your own custom backend that does what you want it to. Just remember that GenEvent is deprecated (to be removed in 2.0) and use :gen_event instead.

[dom]

Yeah, “debug” really is the level you want. The config is not a filter, it changes the level of the response time messages themselves. Other messages (e.g. warnings and errors from Phoenix) will not be affected.

I don’t think filtering per app is a super common problem in practice, because well-behaved Erlang / Elixir libraries provide user hooks rather than log requests directly. Phoenix, Plug and Ecto are all built like that.

[OvermindDL1]

Can’t you just make a filter to filter based on the built-in :module and :application metadata to log whatever you want from different areas?

[cjbottaro]

I want this functionality also.

For example, from Phoenix, I want INFO and higher. From my own library I’m developing, I want DEBUG or higher. From some other library, I want WARN or higher.

Or simply, as a library developer, I want my lib to have it’s own logger that is configured separately from other libs loggers.

[OvermindDL1]

All of these seems pretty easy for what I’ve done anyway.

For example:

error.log: All messages of :error or higher (so just :error) go here.
info.log: All messages of :info or higher go here
wifi.log: All message from the MyServer.WiFI module go here (it handles the local wifi capture portal).
And a few others.

It seems pretty simple though, but I’m not seeing any precise examples of what wants to be done?

[svsdehh]

@OvermindDL1

but I’m not seeing any precise examples of what wants to be done?

I do not know whether you just mean @cjbottaro or me, too.

all our logs go to a central service and wont be saved in different files on the local file system. we use the elastic stack for logging. in the moment we are logging on info and get a view thousand logging entries per minute just from logging every http request and the response time. that does not add any value for us but bloats the log and in the end cost money.

so naturally we would like only to log what has a value for us or can have a value for us. and this is not possible with a global log level. that’s the reason I was asking here for a solution.