Mix Firmware takes *very* long time to run - how to debug?

Longtime erlang guy, but new to nerves. We are trying to get Nerves running for a new project, but are running into very long firmware build times (15+mins) for even very simple example projects. This is with a fresh install of latest versions of elixir/nerves on a powerful macbook pro.

It gets stuck here:
Updating base firmware image with Erlang release…
Copying rootfs_overlay: /Users/…/nerves_test/_build/rpi3_dev/nerves/rootfs_overlay
Copying rootfs_overlay: /Users/…/nerves_test/rootfs_overlay
[15min + delay]

–verbose flags doesn’t provide any more details for these steps.

I assume 15+mins to build blinky (even if we build after no changes have been made) on an MBP isn’t normal. Any pointers on how to solve or debug this are much appreciated.

Given how many Nerves users use MBPs (both x86_64 and M1), I’m surprised this hasn’t come up lately. I can’t recall an issue on slow builds on macs for a couple years. I’m hoping someone else here may see this and respond if they have more clues so that we can get this fixed.

The step that I think that you’re stuck at does a lot of file I/O. Is there anything installed on your laptop that might make this slower?

Also, what version of MacOS are you using?

UPDATE to Below: I’ve narrowed down the issue to merge-squashfs. Digging more…

Thanks - good to know this isn’t expected behavior!

Running fs_usage for my machine over-all and the mix erlang process specifically doesn’t seem to show lots of file I/O. In fact, on the erlang process it shows the opposite - very little happening. Naively, it looks like something is blocking I/O on that process (or just blocking the process in general).

Is there any way to get more verbose outputs for the “Copying rootfs_overlay”? Or other pointers on where I can go digging? I can’t find that string in the nerve-project git org, so I’m guess it’s coming from buildroot? Appreciate the help.

P.S. Running Big Sur on Intel Mac

A couple years ago someone else ran into an issue with merge-squashfs being very slow. I don’t remember it ever being resolved - like it magically went away for them. I think you’re on the right track.

Well - I found how to fix the issue; albeit I can’t explain it. In case anyone else runs across this or can explain why this was the issue:

My $PATH had a reference to a non-existent directory starting with /home/. This caused sed (and other commands, but sed in particular) to load slowly. The scripts behind building the firmware calls sed a lot in a loop.

This obviously isn’t a mac forum, but in case anyone can shed some light on this: If the $path contained non-existent directories that did NOT start with /home/ all was fine. However, having an invalid directory that did start with /home/ dramatically slowed down launching the commands from the bash script. This directory had slipped in from my linux dotfile.

4 Likes

Wow, that is an amazing find. I was able to reproduce a slowdown on my M1 MBP. It was no where near as dramatic as what you experienced (10 second mix firmware time was turned into 40 seconds with a nonexistent /home directory).

I worked around it by caching the absolute path to sed at the top of the script. See nerves_system_br PR.

I feel like there’s more that could be done, but this was so impressively bad that I’d like to get this out quickly. Thank you again for narrowing it down.

2 Likes