Hi,
I’m looking to get as small and fast a firmware as possible for my rpi0 project. I started with an empty project created with --init-gadget
The firmware is about 31MB
100% (31.40 / 31.40) MB
I blink a led as soon as the app is started, and that takes about 19 seconds.
There’s lots of sites/videos out there promoting nerves’ fast boot times (6-8 seconds). Is there any way I can make things faster? Please not that I’m using USB gadget mode for now, but don’t need that for the “final” release of the project. Will it make a big different to strip it?
I’ll also never use HDMI, Sound, 1-wire, Ethernet or bluetooth. I will be using WiFi. Will it make a bit difference when I strip those modules from the linux image?
Unfortunately being single cored, the rpi0 doesnt boot as fast as some of the other Nerves devices. You can trim down quite a few modules but you will still be limited to how fast Erlang can boot. There is also quite a bit of time before the kernel even starts on rpi which you have zero control over. I’ve never tried, but I’m guessing the best time you will be able to achieve is probably 12-15 seconds. (note I’m kind of guessing with that number)
2 Likes
Thanks Connor!
Trying to make sense of the logs here
This is the first line:
00:00:12.640 [info] [ 0.000000] Booting Linux on physical CPU 0x0
Does that mean it takes 12 seconds before it even starts booting linux?
Here some more lines further down:
00:00:12.671 [warn] [ 0.696083] This architecture does not have kernel memory protection.
00:00:12.671 [info] [ 0.899318] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts:
00:00:12.671 [warn] [ 11.666509] heart: kernel watchdog activated (interval 5s)
00:00:12.671 [info] [ 12.601191] random: crng init done
00:00:13.751 [info] Start Network Interface Worker
00:00:13.858 [debug] Elixir.Nerves.Network setup(usb0)
What I don’t understand about this is the difference between the left time and the right. There is big “jump” (0.8....11.6) in the right times, where the left stays the same (00:00:12.671)
1 Like
FYI: I also tried to use a soft sleep mode with a wake button, using Nerves.Runtime.halt() and then waking it again using gpio-shutdown as explain here: https://raw.githubusercontent.com/raspberrypi/firmware/master/boot/overlays/README
It works, but boots just as slowly. I was hoping it would do some sort of hibernation, but behaves just like pulling and plugging the power cord back in.
1 Like
00:00:12.640 [info] [ 0.000000] Booting Linux on physical CPU 0x0
This will usually be the first log from linux, but there are about three processes that happen before this on RPI.
- on first power on RPI boots the GPU
- the GPU for some reason reads the sdcard looking for
bootcode.bin
- the GPU then loads
loader.bin which then loads start.elf which loads linux based on commandline.txt and config.txt on the sdcard.
This takes varying amounts of time on different boards, but id quantify it as about 3-5 seconds before you have any sort of control over how long the device can take to boot. For more information about this i found this Stack Overflow answer
In linux loading the minimum amount of modules INTO the kernel (so CONFIG_SOME_MODULE_OPTION=y instead of CONFIG_SOME_MODULE_OPTION=m) can have some effect. Having a bunch of modules built in with y means it will take longer to unpack the kernel into memory. having a lot of modules built as m means it will take longer at runtime to load them.
You are also at the subject of waiting for the random number generator to come up to set up the ext filesystem for the data partition. If your application doesn’t need to store any data across reboots you can eliminate this. I don’t know exactly how long this takes, but it should only happen on first boot.
Your OTP app (your elixir code) also has to start EVERY one of it’s dependencies before starting. This means if you have a bunch of apps that take forever to start, you will have to wait on them before your app will start.
i’ve noticed on RPI0 there is a good chunk of time just sitting seemingly at idle while BEAM boots. I don’t actually know what is happening, but i suspect it has something to do with only having one core to do things on.
2 Likes
I’ve never tried this before, but i don’t know if RPI supports a soft sleep/suspend state. I’d be interested to see if this is true.
1 Like
When I first started Nerves, the power-on to application running times were about 7.5 seconds. This was on a 720 MHz Beaglebone White (BBW). Given some experience with other embedded Linux products, I felt that I could make it boot even faster if necessary. Many things have happened since then. I’ve been desperately wanting to work on boot time and firmware size again.
First, on measurements, what I did for my measurements was connect the UART of the BBW to my laptop and then I had a program on my laptop that would log a timestamp for each line that was printed to the console. Time zero was the first character out of the UART which was really close to power-on time for that board. I ignored the timestamps reported by the device. Fwiw, the first timestamps that you’re seeing are wall clock timestamps - time starts at 1/1/1970 and it can warp. The kernel timestamps (the second ones in brackets) are monotonic and offset from when time starts for Linux.
Also, don’t measure the first boot after you burn a MicroSD card. The application partition on the card will be formatted, and that is slow (Connor mentioned the random number entropy wait for this which is part of the problem). If you switch the application partition from ext4 to ffs, it’s noticeably faster, but this is a first boot only problem, so probably not worth worrying about.
Before I start speculating, I have the feeling that there are some “hard” things to fix in the initialization. It would be interesting to profile it and have real data.
Barring proper profiling, here are some things that may be low hanging fruit:
(To future people reading this, take these with a grain of salt - we may have fixed them or I might be totally wrong):
-
If you’re not using SystemRegistry for hardware discovery events, turn off the SystemRegistry support in NervesRuntime. I.e. add config :nerves_runtime, :kernel, use_system_registry: false to your config.exs. SystemRegistry gets pummeled by hardware insertion events at boot and is painfully slow on single processor boards. I really hope that this gets fixed.
-
We ended up adding tons of kernel modules and programs to the Raspberry Pi to make it work out of the box better for users. Turn these off. Reducing the number of bytes that needs to be loaded by the RPi’s bootloader will be a small boost. (Totally agree with Connor on this one)
-
If you really don’t need much from the RPi bootloader, the RPi offers at least 3 firmware images: cut-down (cd), default, and extended(x). Nerves uses extended. The other two are smaller and I’ve wondered if they load or initialize faster.
-
We have embedded mode enabled in the rel.vm.args so every beam file is loaded on boot. See http://erlang.org/doc/system_principles/system_principles.html#code-loading-strategy. This ended up helping a couple important Nerves projects a lot so we made it the default. My original measurements were with it off.
-
Per point 4, reducing the number of beam files that get loaded helps when you start pulling in more dependencies. mix tasks often seem to get pulled into projects. Check that runtime: false is listed on your build-time only dependencies. I don’t know if compiling without debug info helps boot time. It definitely helps firmware size.
-
If you have something simple that needs to happen ASAP, add it to the shoehorn init list. OTP applications and their dependencies in the 'init list run first. Just make sure your app doesn’t crash…
-
Switch to faster booting hardware. I really like the Raspberry Pis, but if you have some flexibility with hardware something else will probably be faster. The Raspberry Pi Zero feels slower than it should be compared to other boards of similar clock speed.
If someone is good at profiling the boot sequence, I’d love help getting good data so that if there are any unusual delays that the responsible parties at least are aware of them.
Last, if you are comfortable with embedded Linux, there’s even more that can be done. Many guides will lead you to Buildroot which is pretty convenient since that’s what Nerves already uses. It seems like with embedded Linux systems, there’s always work that can be done to reduce boot times. It’s not unusual to get order-of-magnitude improvements. I know that Nerves can do better.
3 Likes
Geez, thanks a lot for the extensive replies! 
It’s a bit over my head at this time, I’m really just starting with nerves. I did some Arduino stuff in the past (built my own wifi thermostat). I knew boot times would be slower on linux based systems.
As for measuring… I just blink a LED from elixir and use a timer on my phone to measure the time between plugging the power in and seeing the LED come on. I use firmware uploads, and then I turn (un)plug the power so I think I’m not waiting on the formatting to happened that second time around.
To be completely honest: I’m building a portable camera project and I’ve more or less decided to go back to Arduino (ESP32) for this one. Seems to be a better fit. Just a shame I cannot program it in Elixir… I might play around a little with the suggestions provided until my ESP32 dev board arrives. Thanks a lot in any case!
1 Like
Well this may be true for now, but some folks are working on porting an embedded version of BEAM to esp32. https://github.com/bettio/AtomVM it’s nowhere near ready yet, but coming along.
2 Likes
