lawik

lawik

Nerves Core Team

So this is something I am working on for a client I will reference as REDACTED, because they are :slight_smile:

I’ve discussed it a fair bit with Frank and in the interest of sharing knowledge I will try my best to move all that I can in to the open and summarize what got us here. Hopefully we continue our exchanges unhindered and everyone benefits. Perfect world, I get great input. I have not put it in help, it is more of a project.

CM4 is not an ideal security device due to missing the ability to truly protect a secret in hardware (no, the ATECC for NervesKey has limitations here. no ARM TrustZone is on the SoC but has no secured peripherals or storage to use). More on this another time.

But one smart step is to enable secure boot, lock down the bootloader with the OTP (one-time programmable, not our usual OTP) storage part of eeprom shove a public key in there and then sign the boot.img with a boot.sig according to how raspberry pi wants it done.

This is not how fwup works.

fwup does not generate a full .img. It creates a ZIP archive with the .fw extension, signs it. The archive contains all necessary files and a compiled deterministic set of instructions for setting up partitions, copying files and all that. A .fw can be turned into a .img in a simple way.

Nerves generates a .fw. Secure boot according to CM4 methods would mean generating a Nerves firmware as a boot.img and then wrapping it up in a signature and a few additional files to make it boot.

I’ve done this manually and made it work. The question becomes, how do we tool this well in Nerves? It doesn’t really require changing the base build. I would not sign this way in dev, just prod devices.

mix firmware.sign perhaps? It could use fwup to generate a .fw with the boot.img and boot.sig inside it along with some extras. And that means deploying, updating and binary diffs should still be on the table.

I guess the “outer” .fw should own the data partition and so on and the inner .img just the firmware system. Should keep size down.

@fhunleth this a decent summary? Any further thoughts so far?

Showing Posts 1 to 10

fhunleth

fhunleth

Co-author of Nerves

I’ve never tried secure boot on a CM4, so my understanding is from skimming Secure boot chain of trust and the Buildroot configuration added by this commit. Please read with a grain of salt. This could be totally wrong.

This setup looks similar at a high level to secure booting with U-Boot. One difference is that the boot.img file that contains the Linux kernel, device tree, initramfs, etc. uses FAT32 as its container format. So, instead of a ZIP or a tarball or flattened image tree (FIT) file, they chose FAT32. Cool.

The auth flow is:

  1. BootROM validates bootsys and then executes it
  2. bootsys validates bootmain and then executes it
  3. bootmain validates your boot.img and then boots the Linux kernel in it
  4. Your initramfs in boot.img needs to verify the root filesystem and then mount it.

Almost everything in Nerves is for putting together the root filesystem. That’s totally separate from the initramfs so it means that you can create it totally separately if you want. I’d use Buildroot to create it since that’s the tool I know and there’s an example Buildroot config that I linked above. It also works to create a Buildroot package that recursively calls Buildroot so you could automate this in a Nerves system build, but maybe don’t do this first step. First step for me would be to create the boot.img binary manually and download/copy it to the images directory in the Nerves system build.

In your initramfs, you’ll need to mount the root filesystem and switch_root to it. If you google around, I think you should find examples on how to do that. I’d skip the root filesystem auth at this point.

Next step is to go back to Nerves, create a nerves_system_rpi4 fork and update it to know about the boot.img file and the boot.sig file that goes with it. Modify the fwup.conf script to delete all of the normal non-root filesystem blobs that fwup writes and then add the boot.img and boot.sig. The final fwup.conf will write boot.img, boot.sig, and rootfs.squashfs. I’m skipping details, but I think that at a high level this should work and you won’t need to modify any Nerves infrastructure.

After this works, I’d implement whatever mechanism that you want to use to authenticate the root filesystem. It can be anything you want since the code to do the authentication goes in the initramfs which is totally under your control. You may be interested in dm-verity.

The extension that Nerves needs is a hook that you can run after the Nerves tooling creates the read-only SquashFS filesystem and before fwup is called to create the .fw file. The hook would turn the SquashFS filesystem into an image that can be authenticated by dm-verity or whatever you choose to use. After that, fwup would include the result in the .fw and write it to eMMC or MicroSD just like normal. Your initramfs would know that the bytes were a dm-verity’d SquashFS.

I think that it’s important to reiterate that there are multiple keys involved. There are the keys used by CM4 to authenticate the boot.img. You have to do whatever the RPi docs say you have to do for those. Then there are the keys you use to authenticate the root filesystem. You can choose any algorithm for those. You just need to put the public keys that you trust in the boot.img. Maybe you have two boot.img’s that you use - one with the dev public key and one with the prod key so the flow is the same for dev and prod. Up to you.

I hope I understand this right. Interesting project.

lawik

lawik OP

Nerves Core Team

I have made a boot.img from a Nerves BOOT_A and signed it. Then tried booting that using rpiboot from usboot (it doesn’t flash the emmc, just a kind of testboot). And that went fine. So for secure boot itself it is mostly about wrapping up the nerves boot image. Ideally I’d want to do the signing step via fwup so I could have the data partition outside of the boot.img or it will be massive.

The eeprom bootloader stuff I consider fairly separate for now and that is exclusively rpiboot.

I haven’t started on the full disk encryption part as much. Sounds like that will be initramfs and stuff and certainly change things further. Will dig in to what you’ve referenced.

lawik

lawik OP

Nerves Core Team

This is the usbboot of a non-locked secure-boot with a Nerves BOOT_A partition as a boot.img with a boot.sig.

RPi: BOOTLOADER release VERSION:f143c43a DATE: 2024/04/16 TIME: 15:37:20
BOOTMODE: 0x07 partition 0 build-ts BUILD_TIMESTAMP=1713278240 serial 2f2a72cf boardrev b03141 stc 1766378519
Boot mode: RPIBOOT (03) order f25641
secure-boot
Loading boot.img ...
boot.sig
hash: c429ef0fbeba8269bdae601bfc0ec7e1c8f8cb9b2686f9a0c1c3131c2c5fe957
ts: 1713951942
rsa2048: 2b534573150693d45aef4198647315eaef34f260afff921404263362e755fa52dd42ff0f95e6d89dea1ffde0b439f4cc6469a96d07765796e86412d5e99ee35a0e38aeeb28f9d7b23b3f9cfd230bc53730a1f082be744e849142f46cf79e785830064be68dabbbd1dbaf391146a1f59a4d5c172664af72782b0ee58181bfa4191bb25a032602b53078c4d9d210b507fef28a9a30173885cbd1d2726a7ac76cd36ad542316d47a48cff78353bc220e64483ce4e4885e4cf1be403db7119e34e9e7fb5f39fb6e0c9b9a85094383fa3ea52c94668e1a914a177f9ca7830fee0379792532294a239627a7ef3a55f588ff7b2565c275beab885a650316f3d054e6092
Verifying
RSA verify
rsa-verify pass (0x0)
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 16 lba: 0 'MSDOS5.0' '            ' clusters 38295 (1)
rsc 1 fat-sectors 151 root dir cluster 1 sectors 32 entries 512
FAT16 clusters 38295
Read config.txt bytes     1937 hnd 0x16fd
[ramdisk] pieeprom.upd not found
[ramdisk] recover4.elf not found
[ramdisk] recovery.elf not found
Read start4.elf bytes  3002536 hnd 0x14
Read fixup4.dat bytes     8399 hnd 0x3
0x00b03141 0x00000000 0x00001fff
MEM GPU: 192 ARM: 832 TOTAL: 1024
Firmware: 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 Oct 17 2023 15:40:00
Starting start4.elf @ 0xfec00200 partition 0
PCI0 reset
USB-OTG disconnect
+

[    0.587228] brcm-pcie fd500000.pcie: link down
erlinit: The shell will be launched on tty 'tty1'.

erlinit: If you would like the shell to be on this tty,

erlinit: configure erlinit with '-c ttyS0'.

erlinit: The hostname is 'nerves-72cf'.

I assume the things get more interesting and that initramfs comes into the picture when I want an image with a partition scheme I can actually flash to the device. I know we spoke about this during NervesConf. I will try to follow the steps you outlined above and get it going. I expect plenty of confusion :slight_smile:

lawik

lawik OP

Nerves Core Team

This was a bit odd:

==> nerves
==> secure_outer_rpi4
Generated secure_outer_rpi4 app
==> sec_fw

Nerves environment
  MIX_TARGET:   rpi4
  MIX_ENV:      dev

Discovered devices:
0) 0.0 GiB found at /dev/sdb
1) 7.28 GiB found at /dev/sdg
Which device do you want to burn to? 1
 99% [=================================== ] 39.80 MB in / 49.78 MB out       
fwup: Can't open file on FAT partition(boot.sig): FATFS: There is no valid FAT volume

And it does not boot:

RPi: BOOTLOADER release VERSION:e608a69d DATE: 2024/04/15 TIME: 14:12:14
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1713186734 serial 2f2a72cf boardrev b03141 stc 477024
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x1 total-size: 16 Gbit 3200
DDR 3200 0 0 16 152

Boot mode: SD (01) order f2564
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8080 [0]
CID: 001501003847544634520680754ad9ca
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 25000000 Hz actual: 25000000 HZ div: 8 (4) status: 0x1fff0000 delay: 4
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x0000003f,   38630 type: 0x0c
MBR: 0x00012e0c,  289044 type: 0x83
MBR: 0x000a0034,14614476 type: 0x83
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 16 lba: 63 oem: 'MSDOS5.0' volume: '            '
rsc 1 fat-sectors 151 c-count 38295 c-size 1
root dir cluster 1 sectors 32 entries 512
FAT16 clusters 38295
Trying partition: 0
type: 16 lba: 63 oem: 'MSDOS5.0' volume: '            '
rsc 1 fat-sectors 151 c-count 38295 c-size 1
root dir cluster 1 sectors 32 entries 512
FAT16 clusters 38295
secure-boot
Loading boot.img ...
Error 6 loading boot.img
Boot mode: USB-MSD (04) order f256
PCIe timeout: 0x0000008f
USB xHC init failed
Boot mode: NVME (06) order f25
PCIe timeout: 0x0000008f
Failed to open device: 'nvme'
Retry NVME 1
PCIe timeout: 0x0000008f
Failed to open device: 'nvme'
Boot mode: BCM-USB-MSD (05) order f2
XHCI-STOP
xHC ver: 272 HCS: 01000140 0c0000f1 07ff000a HCC: 0220fe65
xHC ports 1 slots 64 intrs 1
USB2[1] 000206e1 connected
USB2[1] 00200e03 connected enabled
USB2 root HUB port 1 init
DEV [01:00] 2.00 000000:01 class 9 VID 0424 PID 2514
HUB init [01:00] 2.00 000000:01
XHCI-STOP
xHC ver: 272 HCS: 01000140 0c0000f1 07ff000a HCC: 0220fe65
USBSTS 18
USB MSD stopped. Timeout: 25 seconds

I have some other stuff to try with the flashing. But not sure what fwup is angry about..

lawik

lawik OP

Nerves Core Team

Ah, their secure-boot-msd is not the way to flash it, probably.

lawik

lawik OP

Nerves Core Team

Misread the fwup error. I’m doing fat_write on something that probably is not fat..

lawik

lawik OP

Nerves Core Team

Because I misread Frank’s post. Lets compile a new one..

lawik

lawik OP

Nerves Core Team

Sometimes, success is a kernel panic:

RPi: BOOTLOADER release VERSION:e608a69d DATE: 2024/04/15 TIME: 14:12:14
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1713186734 serial 2f2a72cf boardrev b03141 stc 477024
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x1 total-size: 16 Gbit 3200
DDR 3200 0 0 16 152

Boot mode: SD (01) order f2564
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8080 [0]
CID: 001501003847544634520680754ad9ca
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 25000000 Hz actual: 25000000 HZ div: 8 (4) status: 0x1fff0000 delay: 4
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x0000003f,   77260 type: 0x0c
MBR: 0x00012e0c,  289044 type: 0x83
MBR: 0x000a0034,14614476 type: 0x83
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 63 oem: 'MSDOS5.0' volume: ' NO NAME    '
rsc 32 fat-sectors 604 c-count 76020 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 76020
Trying partition: 0
type: 32 lba: 63 oem: 'MSDOS5.0' volume: ' NO NAME    '
rsc 32 fat-sectors 604 c-count 76020 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 76020
secure-boot
Loading boot.img ...
SIG boot.sig c429ef0fbeba8269bdae601bfc0ec7e1c8f8cb9b2686f9a0c1c3131c2c5fe957 1713951942
Verifying
RSA verify
rsa-verify pass (0x0)
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 16 lba: 0 oem: 'MSDOS5.0' volume: '            '
rsc 1 fat-sectors 151 c-count 38295 c-size 1
root dir cluster 1 sectors 32 entries 512
FAT16 clusters 38295
Read config.txt bytes     1937 hnd 0x16fd
Read start4.elf bytes  3002536 hnd 0x14
Read fixup4.dat bytes     8399 hnd 0x3
0x00b03141 0x00000000 0x00001fff
MEM GPU: 192 ARM: 832 TOTAL: 1024
Firmware: 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 Oct 17 2023 15:40:00
Starting start4.elf @ 0xfec00200 partition 0
+

[    0.586649] brcm-pcie fd500000.pcie: link down
[    0.881216] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
[    0.889796] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.1.73-v8 #1
[    0.896077] Hardware name: Raspberry Pi Compute Module 4 Rev 1.1 (DT)
[    0.902614] Call trace:
[    0.905095]  dump_backtrace.part.0+0xdc/0xf0
[    0.909451]  show_stack+0x18/0x30
[    0.912824]  dump_stack_lvl+0x64/0x80
[    0.916548]  dump_stack+0x18/0x34
[    0.919914]  panic+0x188/0x344
[    0.923015]  mount_block_root+0x220/0x240
[    0.927092]  mount_root+0x180/0x1a0
[    0.930636]  prepare_namespace+0x124/0x164
[    0.934796]  kernel_init_freeable+0x26c/0x294
[    0.939221]  kernel_init+0x28/0x130
[    0.942761]  ret_from_fork+0x10/0x20
[    0.946389] SMP: stopping secondary CPUs
[    0.950368] Kernel Offset: disabled
[    0.953901] CPU features: 0x80000,00034080,0000420b
[    0.958849] Memory Limit: none
[    0.961948] Rebooting in 10 seconds..
lawik

lawik OP

Nerves Core Team

The boot.img for this one worked to boot via the rpiboot tool but that sidesteps all the initramfs stuff I reckon. It is the BOOT_A partition of a pretty stock rpi4 system.

Next, we init some ram in the fs.

lawik

lawik OP

Nerves Core Team

Where Next? Top

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2977 92995 915
New
caslu
I want to open this thread for you all to discuss and help those who really like Ash but are still hesitant to use it in a real project. ...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
GES233
I’m posting this in response to Jose’s recent tweet (Cr. link) : People are sleeping on Elixir for a coding harness: Hot-code swappi...
New
marciol
It would be helpful to have a list of companies worldwide that hire engineers without prior experience in Elixir. Often, it can be quite ...
New
durvia
Anyone running long-lived stateful processes on BEAM? We’re building an AI agent runtime and would love to compare notes. We’re a small ...
New
AstonJ
This might be a bit disturbing for some but it’s happening - computers running on living human neurons. They’ve made them smart enough t...
New

Other Trending Topics Top

marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
webofbits
Aludel - LLM Evaluation Workbench Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews