Crash in logger (application_terminated,logger,shutdown)

I have been seeing a consistent crash in our system. The erl_crash.dump reports:

Slogan: Kernel pid terminated (application_controller) ({application_terminated,logger,shutdown})

I am running this on the official Elixir alpine docker image, running erlang 22 and elixir-1.9:

System version: Erlang/OTP 22 [erts-10.5.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]

This completely kills the application and there appears to be no way to recover from it. I don’t get any sort of traceback, and the erl_crash.dump doesn’t have any useful information that I can tell. When the crash happens, everything on my node seems normal, memory usage is low, cpu doesn’t spike, there’s no heavy I/O. It is completely random and seems to happen about once a week.

I was hoping when we upgraded to otp22 and elixir-1.9 with logger compile_time_purging, that the issue would go away, but it hasn’t. I am at a loss on where to look.

Thanks in advance!

After a lot of investigation with our ops team, we discovered the issue. We are using a shared docker volume, so that our logger docker sidecar and access the elixir logs and send them to our log management/analytics system. However, the ops teams set the shared volume as a tmpfs (in-memory) filesystem with a max size of 50M.

It appears that we were running out of disk space for the logs, which then caused erlang to crash!

I wonder if there is a nicer way to have erlang’s logger handle this. It certainly would have been nice to at least have a useful message in the erl_crash.dump!

2 Likes