What are your best workflow hacks in Nerves?

Talking to the graybeards and firmware masters over in the Slack for Nerves I’ve gotten some good advice on making my system builds faster. Apparently my instinct a couple of years back to get a Threadripper would have been good. Unfortunately they hadn’t been revised back then and I got a Ryzen 5950X. No slouch but the Threadripper would build faster I hear.

Buildroot defconfig for going faster from Frank and John Ringle:

BR2_JLEVEL=28 # I supposedly have 32 threads so in theory this spares some for my system
BR2_CCACHE=y # This can give a pretty decent speedup for the builds
BR2_CCACHE_DIR=~/.buildroot-ccache
BR2_CCACHE_INITIAL_SETUP="--max-size=1G"

@amclain gave me the idea of putting the ccache on a tmpfs ramdisk. Not sure if it helps but shouldn’t hurt so I did that.

More importantly, while building I get lost in other things and lose useful tens of minutes not knowing when the build is done. I’ve currently bash-scripted my weird multi-step system build (see other thread about secure boot). This run I’m trying out espeak-ng to tell me when build steps complete. Along with /usr/bin/time -o bla-time.txt ./bla-rebuild.sh for every step of the build to get an idea of time elapsed.

What do you do to make life easier when building custom system images frequently?

3 Likes

What are your best workflow hacks in Nerves?

We’ve created the Redwire Labs Nerves Guide to share our favorite tips and tricks for Nerves firmware development. It’s an opinionated guide that favors minimizing development time and shipping features, which is important for our commercial work. Reducing equipment cost is an anti-goal, so this guide is not for every project.

The guide is currently released as a community preview on GitHub. It will have a website eventually.

Buildroot defconfig for going faster from Frank and John Ringle

A side note for context: top-level parallel builds are explained in the Buildroot manual.

BR2_PER_PACKAGE_DIRECTORIES also needs to be set for a parallel package build, and the build needs to be run with make -j. If the number of processors needs to be determined across different machines, BR2_JLEVEL can be set to 0 to auto-detect all of the processors if headroom for other apps is not required.

BR2_JLEVEL=0
BR2_PER_PACKAGE_DIRECTORIES=y
make -j

I experimented with parallel builds a few years ago with nerves_system_bbb since I was working with the TI AM335 a lot at the time. The issue with parallel builds is that something seemed to be wrong with the dependency tree in Buildroot because it would fail part way through the build, I’d restart it, it would get a little further and fail, and so on. BBB builds would take around 3 minutes doing it this way, but it was tedious to have to keep an eye on it and easier to wait the 9-ish minutes for a serial build.

Fast forwarding to now, that issue seems to be sorted out. We have been exploring turning on parallel builds for the AM62 because it takes a significant amount of time to compile, even on a Threadripper (just over 20 minutes). For the SAMA5D27 it’s a nice-to-have.

Here are some examples of compile times:

nerves_system_sama5d27_wlsom1_ek

Serial build
real    11m14.971s

Parallel cold cache
real    2m56.408s

Parallel warm cache
real    2m54.837s

----------

nerves_system_bbb

Serial build
real    11m48.778s

Parallel cold cache
real    2m44.667s

Parallel warm cache
real    2m15.226s

@amclain gave me the idea of putting the ccache on a tmpfs ramdisk. Not sure if it helps but shouldn’t hurt so I did that.

More sidenote first: Wendel from Level One Techs has great videos on benchmarking all sorts of systems, explaining the underlying technology, and showing where performance differences come from. One video that is very close to building Nerves systems is the build server for Greg Kroah-Hartman (Linux kernel maintainer).

So back to the topic: Build cache helps, but the processor is the most important piece of hardware. Memory is next. So if either of these aren’t great, it’s going to choke the system. Turn off the swap file on a build machine because that’s creating really slow memory, and the build doesn’t know if it’s using it or not.

I said to set CCACHE_DIR to a RAM disk (tmpfs) because memory is going to be faster than disk access, but that was the short answer for you to play with. The longer explanation is that we’ve been considering moving the Nerves artifact completely onto a RAM disk when compiling it. This is how that changes the BBB numbers:

RAM disk build cold cache
real    2m36.431s

RAM disk warm cache
real    2m9.249s

Moral of the story: Moving stuff to RAM shaves some time off, but it’s more of an optimization. Buy the best processor you can first. We’ve explained processor selection in the Nerves Guide.

I’ll ping Abelino to see if he has any more thoughts to add since he’s also involved in a lot of this research.

4 Likes

Awesome stuff Alex :slight_smile: