Do OTP logger handlers have overloading protection or not?

Questions / Help
,
1

Hiya. I’m trying to switch from LoggerFileBackend to Erlang/OTP Handlers for logging. In the Elixir Logger docs, it says:

At the moment, there is no built-in overload protection for Erlang handlers, so it is your responsibility to implement it.

However, in the Erland docs it says:

The default handlers, logger_std_h and logger_disk_log_h, feature an overload protection mechanism, which makes it possible for the handlers to survive, and stay responsive, during periods of high load (when huge numbers of incoming log requests must be handled)

This sounds a bit ambiguous to me. Would anyone be able to clarify this?
Thanks!

2

The elixir library doesn’t implement overload protection, however if you use erlang logger backend, the backend will have overload protection?

Does that make sense? I am not entirely familiar with logger backends, but it seems that should be the principle.

1 Like
3

Thanks for the quick reply, it does make a bit more sense.

However, I first read “built-in” as meaning “there is no built-in overload protection in Erlang” and not as “there is no built-in overload protection in Elixir anymore.”

You know what I mean?

4

Yeah, the choice of wording is strange indeed, maybe there is something more to it.

It might also be that erlang did not have overloading protection back in the day when this was written down in elixir documentation, but this is just a theory.

I guess you have to wait for a elixir core member to answer.

1 Like
5

Thanks, @D4no0 , I appreciate the effort.
I’ll wait for a core member to answer.

6

also, even if you’re using the erlang backend you should test what happens when you go overboard. I think we had to change the settings to accomodate the issue when a log drain on Pod was blocking, and with the default settings the Erlang/Phoenix app still locked up because of this completely. This was not the most recent OTP and I know there were logger changes but you should not just rely on the fact that it has overload protection, you should check how it works and behaves in your set up and it’s one of the things you may want to tweak otherwise the total system collapse is very much possible :wink:

7

Is it truly that bad? When you drain requests from a pod that’s shutting down that is (unless I misread you)?

8

Well, yes, overload protection and ensuring the handlers survive is one thing, the other thing is how the system behaves and if it survives when the output of logs exceedes the capacity to process them.

Logger has several options to handle that situation, the ones you may want to tweak are :sync_threshold, :discard_threshold and :max_buffer.

Basically we had situation where logger was switching to sync mode, and stayed in it too long, the I/O was stuck and we just decided never to do it because the whole system would freeze, and switched to discard mode instead of ever hitting sync mode was the best option for us.

2 Likes
9

Good to know, thanks for indulging. I suppose I was under the illusion that having a maximum buffer and discarding under load were the defaults.

10

Thanks for that conversation, @dimitarvp and @hubertlepicki. It definitely sounds like discarding under load and having a maximum buffer should be defaults.

In any case, going back to the original question, it does sound like handlers DO have some overload protection and one needs to test how to tweak it.

11

If I am reading the documentation right, the Logger will switch to sync mode after 20 messages in the buffer, then will wait for 480 more messages before it switches to discard mode. On the nodes that are not doing a lot of logging that means these requests are stuck for that time.

https://hexdocs.pm/logger/1.12.3/Logger.html

1 Like
12

I think it is the default but you should still simulate this to see how resilient the system is and if/what you have to tweak.

13

The docs you referred to are for 1.12.3, but we’ll be going to 1.16.1, so here:
https://hexdocs.pm/logger/1.16.1/Logger.html

The section in question is the one about Erlang/OTP handlers:

https://hexdocs.pm/logger/1.16.1/Logger.html#module-erlang-otp-handlers

which states that:

At the moment, there is no built-in overload protection for Erlang handlers, so it is your responsibility to implement it.

1 Like