When I first started Nerves, the power-on to application running times were about 7.5 seconds. This was on a 720 MHz Beaglebone White (BBW). Given some experience with other embedded Linux products, I felt that I could make it boot even faster if necessary. Many things have happened since then. I’ve been desperately wanting to work on boot time and firmware size again.
First, on measurements, what I did for my measurements was connect the UART of the BBW to my laptop and then I had a program on my laptop that would log a timestamp for each line that was printed to the console. Time zero was the first character out of the UART which was really close to power-on time for that board. I ignored the timestamps reported by the device. Fwiw, the first timestamps that you’re seeing are wall clock timestamps - time starts at 1/1/1970 and it can warp. The kernel timestamps (the second ones in brackets) are monotonic and offset from when time starts for Linux.
Also, don’t measure the first boot after you burn a MicroSD card. The application partition on the card will be formatted, and that is slow (Connor mentioned the random number entropy wait for this which is part of the problem). If you switch the application partition from
ffs, it’s noticeably faster, but this is a first boot only problem, so probably not worth worrying about.
Before I start speculating, I have the feeling that there are some “hard” things to fix in the initialization. It would be interesting to profile it and have real data.
Barring proper profiling, here are some things that may be low hanging fruit:
(To future people reading this, take these with a grain of salt - we may have fixed them or I might be totally wrong):
If you’re not using SystemRegistry for hardware discovery events, turn off the SystemRegistry support in NervesRuntime. I.e. add
config :nerves_runtime, :kernel, use_system_registry: false to your
config.exs. SystemRegistry gets pummeled by hardware insertion events at boot and is painfully slow on single processor boards. I really hope that this gets fixed.
We ended up adding tons of kernel modules and programs to the Raspberry Pi to make it work out of the box better for users. Turn these off. Reducing the number of bytes that needs to be loaded by the RPi’s bootloader will be a small boost. (Totally agree with Connor on this one)
If you really don’t need much from the RPi bootloader, the RPi offers at least 3 firmware images: cut-down (
cd), default, and extended(
x). Nerves uses extended. The other two are smaller and I’ve wondered if they load or initialize faster.
We have embedded mode enabled in the
rel.vm.args so every beam file is loaded on boot. See http://erlang.org/doc/system_principles/system_principles.html#code-loading-strategy. This ended up helping a couple important Nerves projects a lot so we made it the default. My original measurements were with it off.
Per point 4, reducing the number of beam files that get loaded helps when you start pulling in more dependencies.
mix tasks often seem to get pulled into projects. Check that
runtime: false is listed on your build-time only dependencies. I don’t know if compiling without debug info helps boot time. It definitely helps firmware size.
If you have something simple that needs to happen ASAP, add it to the
init list. OTP applications and their dependencies in the
'init list run first. Just make sure your app doesn’t crash…
Switch to faster booting hardware. I really like the Raspberry Pis, but if you have some flexibility with hardware something else will probably be faster. The Raspberry Pi Zero feels slower than it should be compared to other boards of similar clock speed.
If someone is good at profiling the boot sequence, I’d love help getting good data so that if there are any unusual delays that the responsible parties at least are aware of them.
Last, if you are comfortable with embedded Linux, there’s even more that can be done. Many guides will lead you to Buildroot which is pretty convenient since that’s what Nerves already uses. It seems like with embedded Linux systems, there’s always work that can be done to reduce boot times. It’s not unusual to get order-of-magnitude improvements. I know that Nerves can do better.