teqr

Logging stops completely due to Logger crash

This question might dive into sub-areas, so I would like to clarify as much as possible.

Application:
The application has multiple genserver process started though Supervisor and Dynamic supervisor. Application also uses elixir Logger, LoggerFileBackend to log into different files.

Behaviour:
Sometimes some of the process crash also (and recover successfully). Sometimes Logger process crashes completely and logging in all the files stops. This is completely random, happening at stretch of months, or sometimes multiple times in a day. There is no pattern or logs which might indicate the reason and it is not recoverable (for the logger process). However, the application runs fine (for months without logging) and there are no logs or any crash report during this event. This issue only gets resolved after restarting the application (which is not acceptable).

Observations:
As there are no log reports, so this is based on debugging which I was able to do:

From application iex shell, I can confirm that Process.whereis(Logger) returns nil.
I assume that some genserver might be crashing multiple times repeatedly - causing Logger to crash entirely. The application genserver is able to recover, however, the logger doesnt recover. This is only my assumption as I dont get any crash report for any genservers of application during this event. Other genserver crashes do get logged properly (when Logger has not crashed).

Environment:

FROM erlang:25.1-alpine
ENV ELIXIR_VERSION="v1.13.4" LANG=C.UTF-8

extra_applications: [:lager, :logger, :runtime_tools]

My questions:

Is there any better way to configure logs, so as to identify the reason for Logger crashes?
Assuming due to some reason, Logger crashes inside my application. Is there any way to restart it (without restarting the complete application)?

5 comments

#logger #genserver #logger_file_backend

0 548 5

2024-06-13 20:10:57 UTC

Most Liked

dimitarvp

Flying blind would worry me. As a first measure I’d do my best to find why, even if it takes a while. As a second measure I’d fire up a separate process that periodically checks if the logger is alive and try to start it if it’s dead; I reckon once every 5 seconds is enough.

But really, don’t you get any stack traces? If you configure f.ex. Loki then all the stdout and stderr outputs get put in the logging backend of Grafana and you’ll see everything.

Post #2

LostKobrakai

Enabling sasl logging might also help depending on which version of elixir/otp you’re running

Post #3

hauleth

That is not surprising, as there is no such process at all in Logger. It seems that the Logger in :registered_names in logger.app file is just an omission. You can easily test that by running:

$ iex
iex> Process.whereis(Logger)
nil

My bet there is that either there is some message that handler was removed because of some failure, or there is message that there is too many messages in the process inbox and you run in the drop mode.

Post #4