Phoenix 1.8 + Nerves on Pi 5 B boot loop

Nerves Forum Nerves Questions / Help
, ,
1

Hello astute Nerves community,

I am struggeling to get a Pi 5 running with Phoenix 1.8.1 and Nerves. I have a working Phoenix 1.7.2 system running, and have upgraded my “Nerves + Phoenix 1.7” Branch to 1.8 using a diff as describes in this topic.

Everthing compiles fine (I deleted the _build and deps directories beforehand) but after inserting the burned SD card, my Pi gets stuck in a reboot loop. The LED signals first red after boot, after 1 second it turns into a solid green, at around 3 seconds after boot the fan starts up. 5-6s after boot the green led starts to blink rapidly and at around 8s it reboots, (fan turns off, LED turns red) and the cycle starts again.

I tried to connect an (kinda old) LCD monitor via HDMI, but unfortunately the error (and reboot) manifests itself to soon before the screen stabilizes and shows any output, therefore it only starts up from standby/sleep, recognizes the input signal, blinks, and then get into standby again. :frowning:

I switched back to my 1.7.2 branch and the pi starts up again, so its clearly a software issue. My diff looks pretty much as the one mentioned above (adopted for the app name), as my 1.7.2 branch is from a stock nerves.new and phx.new output without any modifications.

Any pointers or hints are most welcome.

Thanks

Peter

2

I’m still struggeling, so I did what I always do in situations like this, I read the very fine :heart: manual.

I did add verbose: true, hang_on_fatal: true, run_on_exit: “/bin/sh” to the erlinit config, but that did only result in a subtle change. The screen stays active some seconds longer before entering sleep, but still no output.

I did boot the Pi without the SD card, I can see the Raspberry bootloader info on the screen. If I then insert the card, it finds the kernel and boots it, then a blank screen for a while and than reboot.

I double checked, that my upgrade did not change any nerves packages, only phoenix related stuff.

Will investigate further…. :nerd_face:

3

The in-depth thing to do would be to get a UART console via an FTDI cable so you can ser the boot log properly. But that’s a bit of a nuisance if you don’t have it.

Does it require custom hardware to run or could you try adding the nerves_system_qemu_aarch64 package? If that exhibits the same issue you should be getting the full log in your terminal..

3 Likes
4

If it’s Phoenix related, maybe one thing you can try is to not start your endpoint in your application tree. Then, if it works and your app doesn’t crash on boot, you can call MyAppWeb.Endpoint.start_link at the iex terminal. That should get you error messages at least. If that doesn’t work, it would be more effort but you could start making all the Phoenix-related deps as runtime: false. And then you would have to start them manually before starting your endpoint. The easiest path is definitely the UART-USB cable - if you can get one, it is very useful for almost any embedded project.

I’ll also throw out here that if you don’t have a UART-USB cable, but you have one of various types of popular microcontroller boards, you can probably find a firmware that someone has made to do the conversion.

For example, if you have a Raspberry Pi Pico, you could wire up GitHub - Noltari/pico-uart-bridge: Raspberry Pi Pico UART-USB bridge to get the UART console. I’m sure other microcontroller dev boards have similar firmwares if you go looking for them (keywords that helped me find this repo: “rpi pico uart to usb”).

But yeah. Since it’s crashing on boot it’s tricky to debug without the cable

5 Likes
5

Hi Lars,

thanks for the pointer, I do have an FT232 board lying around, but I do seem to miss a JST SH connector for the UART interface of the RPi5.

Will try to get one tomorrow, I tried to salvage some spare/broken/old electronics a have but no luck in finding one.

As for the qemu path, I first want to explore the hardware path, but it also sounds intriguing, so I will try anyway…

6

Hi Gus,

thank you for your answer and suggestions, I tried not starting phoenix and also tried with runtime: false but I do get the same behaviour (reboot after some seconds).

I do have some spare Picos lying around, but currently I’m missing a JST SH connector, will try tomorrow again.

7

Why wait for hardware, If I can have it in software…

Gone down that qemu rabbit hole, for the record, I added

{:nerves_system_qemu_aarch64, “~> 0.1.1”, runtime: false, targets: :qemu} to mix.ex,

built the firmware with

MIX_TARGET=qemu MIX_ENV=prod mix do deps.get, firmware and generated the command with MIX_TARGET=qemu MIX_ENV=prod mix nerves.gen.qemu _build/qemu_prod/nerves/images/my_app.fw

And voila, I can see the error message! Many thanks to the whole Nerves team!

(I had to kill -9 qemu, because it looped forever, without accepting any keyboard input.)

Erlang/OTP 27 [erts-15.2.7.2] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit]

ERROR! the application :my_app has a different value set for path [:code_reloader] inside key MyAppWeb.Endpoint during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behavior than expected:

  * Compile time value was not set
  * Runtime value was set to: false

To fix this error, you might:

  * Make the runtime value match the compile time one

  * Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.clean my_app --build"

  * Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag


heart: Erlang is crashing .. (waiting for crash dump file)
heart: waiting for dump - timeout set to 5 seconds.
Runtime terminating during boot ({<<"aborting boot">>,[{'Elixir.Config.Provider',boot,2,[]}]})

Crash dump is being written to: /root/erl_crash.dump...
[nbtty: terminating]
erlinit: Erlang VM exited
erlinit: run_cmd '/bin/sh'
# heart: Unable to kill old process, kill failed (tried multiple times):  Success
reboot: Restarting system
little_loader 0.1.1

So of course its my config messup, which I fixed, and I know now how to debug any future botches.

Many thanks and a big :heart: for the Nerves core team and community!

3 Likes
8

That was the idea with the qemu suggestion, just to get at the boot process. Glad it worked :slight_smile:

1 Like
9

Thank you Lars for your suggestion, I would not come up with that idea by myself.

1 Like