What are your best workflow hacks in Nerves?
We’ve created the Redwire Labs Nerves Guide to share our favorite tips and tricks for Nerves firmware development. It’s an opinionated guide that favors minimizing development time and shipping features, which is important for our commercial work. Reducing equipment cost is an anti-goal, so this guide is not for every project.
The guide is currently released as a community preview on GitHub. It will have a website eventually.
Buildroot defconfig for going faster from Frank and John Ringle
A side note for context: top-level parallel builds are explained in the Buildroot manual.
BR2_PER_PACKAGE_DIRECTORIES also needs to be set for a parallel package build, and the build needs to be run with make -j. If the number of processors needs to be determined across different machines, BR2_JLEVEL can be set to 0 to auto-detect all of the processors if headroom for other apps is not required.
BR2_JLEVEL=0
BR2_PER_PACKAGE_DIRECTORIES=y
make -j
I experimented with parallel builds a few years ago with nerves_system_bbb since I was working with the TI AM335 a lot at the time. The issue with parallel builds is that something seemed to be wrong with the dependency tree in Buildroot because it would fail part way through the build, I’d restart it, it would get a little further and fail, and so on. BBB builds would take around 3 minutes doing it this way, but it was tedious to have to keep an eye on it and easier to wait the 9-ish minutes for a serial build.
Fast forwarding to now, that issue seems to be sorted out. We have been exploring turning on parallel builds for the AM62 because it takes a significant amount of time to compile, even on a Threadripper (just over 20 minutes). For the SAMA5D27 it’s a nice-to-have.
Here are some examples of compile times:
nerves_system_sama5d27_wlsom1_ek
Serial build
real 11m14.971s
Parallel cold cache
real 2m56.408s
Parallel warm cache
real 2m54.837s
----------
nerves_system_bbb
Serial build
real 11m48.778s
Parallel cold cache
real 2m44.667s
Parallel warm cache
real 2m15.226s
@amclain gave me the idea of putting the ccache on a tmpfs ramdisk. Not sure if it helps but shouldn’t hurt so I did that.
More sidenote first: Wendel from Level One Techs has great videos on benchmarking all sorts of systems, explaining the underlying technology, and showing where performance differences come from. One video that is very close to building Nerves systems is the build server for Greg Kroah-Hartman (Linux kernel maintainer).
So back to the topic: Build cache helps, but the processor is the most important piece of hardware. Memory is next. So if either of these aren’t great, it’s going to choke the system. Turn off the swap file on a build machine because that’s creating really slow memory, and the build doesn’t know if it’s using it or not.
I said to set CCACHE_DIR to a RAM disk (tmpfs) because memory is going to be faster than disk access, but that was the short answer for you to play with. The longer explanation is that we’ve been considering moving the Nerves artifact completely onto a RAM disk when compiling it. This is how that changes the BBB numbers:
RAM disk build cold cache
real 2m36.431s
RAM disk warm cache
real 2m9.249s
Moral of the story: Moving stuff to RAM shaves some time off, but it’s more of an optimization. Buy the best processor you can first. We’ve explained processor selection in the Nerves Guide.
I’ll ping Abelino to see if he has any more thoughts to add since he’s also involved in a lot of this research.
