Hey guys! Here is something I’m thinking lately: why not setting config :logger, level: :debug in prod env?
The point would be to have more information about errors that happened and are hard to repeat, without having to create a specific new log message, deploy it to prod and crossing your fingers the info would be enough when the error happens again. The biggest con I can see, ofc, is performance. Do any of you use debug logger level in prod? Do we have any benchmark for it? And is that actually a problem for the end users? I mean, IO should not block the request cause the logger is asynchronous, right? I guess it should not impact the request as much in that case right?
I remember setting the log level to debug on other prod envs in Ruby, and acknowledging ruby is a lot slower than Elixir, and even worse, logs on ruby are not asynchronous, I can’t see the point of leaving it on the info level.
Debug logs can contain private informations that should not be stored in the log system. For example email can be considered PII, which mean that your logs now require to be able to comply with GDPR (assuming you are operating in EU) and you need to provide and remove that informations on request, including historical data.
Overloading logging system. Debug logs are often defined to be very descriptive and can contain a lot of information, as well as fire quite often. This can overload logger and then cause backends/handlers to just drop them on the flor to prevent system from DoS.
Signal to noise ratio. Debug logs can be quite noisy and can hide important and relevant information in SQL queries and stuff like that.
Network transfer in case of centralised logging systems. Mostly the same points as in 2., but with different place where the slowdown/congestion can happen.
Thanks for all your points. I agree all of them are a concern you have to have in mind, but I really am trying to measure the pros vs cons, and as I said, I’m really missing some of the info I used to have on rails logger. So here are my points about each of them:
True, that might be a problem, but solvable by hiding the data like it’s done to passwords on ecto and phoenix debug logs, but in my mind logs are only accessible by us, so we don’t have many problems with that. The biggest issue then would be having to remove the user data from logs when requested, but that is also not impossible, just have to go through log files and delete the data.
Kinda related to my performance comments: I think the BEAM is reliable enough to trust that if the logger becomes a problem itself, the system will recover from that and we have time to work on a fix for it.
Agree, there will be a lot of stuff that we are not going to use, and we actually don’t want to use either. But if we go that direction, then some of the info level logs also can be considered noise. I mean, it’s noise because it’s not useful enough to actually get more information about what happened. The thing is that we actually don’t know what will be noise, so I would prefer 10 times to have a lot of noise and some signal between it, than not having signal at all (which happened multiple times for me already)
Normally when centralizing logs we still leave the old log files on the server anyway, so I guess we have a backup if that becomes a problem. We can live while solving the problem.
Until you will have breach, and your logs will be read not only by You.
Then you probably use wrong log level for that data.
Not really. BEAM can actually be a problem there as sending messages to processes is cheap and have negligible backpressure. Unbounded growth of the mailbox can cause VM to OOM and degrade performance of all processes. That is why Elixir, Erlang, and Lager implement some kind of overload protection mechanisms. Additionally that is the reason why Elixir do not encourage people to use “Erlang handlers” instead of “Elixir backends”, as there is no reliable way to have such mechanism in Erlang handlers yet.
Yes, that is why Erlang use notice as a default level, and Elixir currently uses some hacks to work around it for compatibility reasons.
Debug logs are meant for debugging. If you want to have them enabled by default in production, then you have used wrong level in the first place. So you are looking wrong way at the problem - make interesting logs for you use higher log level instead of lowering primary log level.
Yeah, in the end it just points to a lack of interest I have on actually making the current log level a bit better You’re right, debug should be for debugging and that’s it.
Bugs happen in production. So that’s why one would want information to debug it.
I’m responding because I’ve had the same idea. Mostly when a bug happens and I can’t tell what actually happened from the logs. Even just a stacktrace would be nice.
You’re right, debug should be for debugging and that’s it.
