Accessing System-Level Logs for Debugging Spontaneous Shutdowns on Raspi Zero

I’m currently working on a project with Nerves(Nerves Livebook) on a Raspberry Pi Zero, and I’ve encountered a recurring issue where the device spontaneously shuts down. To better understand what’s happening behind the scenes and potentially pinpoint the cause, I’m looking to dive into the system-level logs.

Could anyone guide me on how to access these logs within the Nerves environment? Specifically, I’m interested in any logs that could shed light on system events or errors that might precede these unexpected shutdowns.

Debugging spontaneous shutdowns can be hard since the key log message might not be logged.

There are a few options:

  1. If you’re using RingLogger, turn on persistence. RingLogger will write logs every so many minutes and then on graceful shutdown. Not sure if this will help given what you posted, but seems worth a try. In your config, do something like this:
config :logger, RingLogger,
  persist_path: "/data/ring_logger.log",
  persist_seconds: 300,
  1. Use RamoopsLogger. This backend logs to a special area of DRAM that survives reboots sometimes. Add the dependency and add RamoopsLogger to your :logger :backends in the config.exs.

  2. Enable the :console logger, connect to the UART, watch messages and hope.

  3. Enable an Elixir or Erlang file logger like you would for a non-Nerves project. Configure it to log to a file in /data. There might be a sync option to force writes more frequently. I don’t personally use this option, but I know people coming from non-embedded backgrounds sometimes are really comfortable with this one.

On Nerves, log messages from the Linux kernel and from C programs using syslog are all routed through the Elixir logger. Hopefully these aren’t your issue.

Some of the hardest bugs, imho, to debug are the ones that kill heart. If this is your issue, then try running Nerves.Runtime.Heart.status to see if wdt_time_left or heartbeat_time_left count down to 0. When they hit 0, heart thinks that something is really wrong with the BEAM or something else on the device and triggers a reboot.

Hope this helps.

1 Like