fireindark707

fireindark707

Accessing System-Level Logs for Debugging Spontaneous Shutdowns on Raspi Zero

I’m currently working on a project with Nerves(Nerves Livebook) on a Raspberry Pi Zero, and I’ve encountered a recurring issue where the device spontaneously shuts down. To better understand what’s happening behind the scenes and potentially pinpoint the cause, I’m looking to dive into the system-level logs.

Could anyone guide me on how to access these logs within the Nerves environment? Specifically, I’m interested in any logs that could shed light on system events or errors that might precede these unexpected shutdowns.

Marked As Solved

fhunleth

fhunleth

Co-author of Nerves

I think the first thing to try is to just disable heart altogether. To do this, edit your project’s rel/vm.args.eex and comment out the -heart line:

## Enable heartbeat monitoring of the Erlang runtime system
#-heart -env HEART_BEAT_TIMEOUT 30

I really hope that gives you time to look at log, inspect your device or do whatever to see why Erlang isn’t telling heart that it’s ok, and therefore not petting the hardware watchdog.

As for changing the hardware watchdog timer, it’s already set to the longest time. If you trace through bcm2835_wdt.c, the max time looks like it’s just under 16 seconds and that due to truncation, it shows up as 15 seconds when asked.

Best bet is to comment out heart like above. I don’t know how 30 seconds could be too short on the heart timeout, but you could also update that.

Hope you find the issue.

Also Liked

fhunleth

fhunleth

Co-author of Nerves

Debugging spontaneous shutdowns can be hard since the key log message might not be logged.

There are a few options:

  1. If you’re using RingLogger, turn on persistence. RingLogger will write logs every so many minutes and then on graceful shutdown. Not sure if this will help given what you posted, but seems worth a try. In your config, do something like this:
config :logger, RingLogger,
  persist_path: "/data/ring_logger.log",
  persist_seconds: 300,
  1. Use RamoopsLogger. This backend logs to a special area of DRAM that survives reboots sometimes. Add the dependency and add RamoopsLogger to your :logger :backends in the config.exs.

  2. Enable the :console logger, connect to the UART, watch messages and hope.

  3. Enable an Elixir or Erlang file logger like you would for a non-Nerves project. Configure it to log to a file in /data. There might be a sync option to force writes more frequently. I don’t personally use this option, but I know people coming from non-embedded backgrounds sometimes are really comfortable with this one.

On Nerves, log messages from the Linux kernel and from C programs using syslog are all routed through the Elixir logger. Hopefully these aren’t your issue.

Some of the hardest bugs, imho, to debug are the ones that kill heart. If this is your issue, then try running Nerves.Runtime.Heart.status to see if wdt_time_left or heartbeat_time_left count down to 0. When they hit 0, heart thinks that something is really wrong with the BEAM or something else on the device and triggers a reboot.

Hope this helps.

fireindark707

fireindark707

Thanks @fhunleth , I have figured out the problem. It relates to a problem of Kino. I have report the issue:

https://github.com/livebook-dev/kino/issues/416

Where Next?

Popular in Questions Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42158 114
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
New

We're in Beta

About us Mission Statement