Allow to specify image size when using mix firmware.image

Nerves Forum Nerves Questions / Help
, ,
1

Hey there!

This is more of a feature requests to the people knowing the internals of fwup. I’m currently testing some nerves systems in QEMU, therefore I’m using mix firmware.image to write a raw disk image. The problem is that, compared to a real device, the application partition is never expanded to a reasonable size. To workaround this, I currently need to

  1. resize the image (qemu-img resize disk.img +20G)
  2. boot up the image once to create the filesystem
  3. stop the machine again and boot into the gparted live system
  4. resize the partition in gparted
  5. stop the machine and eject the gparted live iso
  6. finally boot up nerves with a reasonable app partition size

I tried to look into the fwup internals, but that’s out of my comfort zone…
I’m imagining something like mix firmware.image disk.img --size 20G to create an image with 20GB in size where the application partition expands like when burning a 20GB SD card. This is probably something that would need to be implemented in fwup itself, e.g. fwup -a -i firmware.fw -t complete -d disk.img --size 20G.

Thanks!

1 Like
2

What you’re doing sounds like the normal path outside of qemu, but just not automated.

Before I expand on this, if you’re only going to be working in qemu with a 20GB application partition, I’d change the fwup.conf to specify a 20GB partition. To do this, change the line that looks like this:

define(APP_PART_COUNT, 524288)

to

define(APP_PART_COUNT, 42949672960) # 20GB in units of 512-byte blocks

Normally the fwup.conf files specify a lowest expected size of the destination media. This is for practical reasons since there’s variation in 4GB eMMC parts or maybe you want to accommodate whatever MicroSD or eMMC that is the easiest to buy or maybe you want flash programming to go faster on the manufacturing line.

Regardless of the reason, the application data partition needs to be formatted on the first boot. Nerves does that for you in Nerves.Runtime. However, you could change that code to run gparted to max out the application data partition first and then do the first-time format. This would save the reboot and the manual hassle that you’re going through.

Sadly, we’ve never done this with Nerves. In hindsight, I’m a little surprised that the need hasn’t arisen, but I’m not sure many Nerves users need to max out the application data partition. Or more likely, the change was too device-specific and never contributed upstream. I hope that in your case, adjusting the fwup.conf might be “good enough”.

2 Likes
3

I don’t want to make assumptions about the minimum size, as QEMU is really just for testing.

I tried to do the following

$ qemu-img create disk.img 20G
$ MIX_TARGET=x86_64 mix firmware.image disk.img

but sadly, fwup still doesn’t expand the application partition. I assume it doesn’t do any expansion when it targets an image - but that sounds like something that should not be too hard to implement, right?

If that’s not possible, I might just set the minimum size to 2GB or something :smiley:

4

fwup can expand the final partition to the size of the media that its targeting using expand = true in the partition description. Most Nerves systems use this option including nerves_system_x86_64. See nerves_system_x86_64/fwup.conf at main · nerves-project/nerves_system_x86_64 · GitHub.

The trick is that fwup has to be able to determine the size of the destination. If it can’t, then you can pass the --max-size <512-byte blocks> option to fwup.

Your first line of creating the 20G disk.img should create a file that fwup can figure out the size for automatically. I just looked at the code and I don’t see why this wouldn’t work (assuming expand = true on the final partition). If it doesn’t, could you post an issue with the fwup.conf or a simplified version that doesn’t work so I can replicate?

5

Indeed, expand = true: nerves_containers_x86_64/fwup.conf at 0da6d8b483b568a5353c681ef0545dc0c9d6a574 · nerves-containers/nerves_containers_x86_64 · GitHub

Nice, that sounds like exactly what I need! But trying it, it doesn’t work:

$ qemu-img create disk.img 20G
$ fwup -a -d disk.img -i _build/x86_64_dev/nerves/images/docker_example.fw  -t complete --max-size 42949672960

Looking at the resulting image, the MBR did not expand :thinking:

$ hdiutil attach disk.img -nomount
/dev/disk4          	FDisk_partition_scheme
/dev/disk4s1        	Windows_FAT_32
/dev/disk4s2        	Linux
/dev/disk4s3        	Linux
/dev/disk4s4        	Linux
$ fdisk /dev/disk4
Disk: /dev/disk4	geometry: 2610/255/63 [41943040 sectors]
Signature: 0xAA55
         Starting       Ending
 #: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
*1: 0C    0  65   2 -    2  50  48 [      4096 -      31232] Win95 FAT32L
 2: 83    2  50  49 -   34 212  50 [     35328 -     524288] Linux files*
 3: 83   34 212  51 -   67 119  52 [    559616 -     524288] Linux files*
 4: 83   67 119  53 -  100  26  54 [   1083904 -     524288] Linux files*
$ hdiutil detach /dev/disk4

You can use the example repo I’m currently testing with, all the systems are prebuilt: GitHub - nerves-containers/docker_example: Example project using the nerves_containers base systems

$ git clone https://github.com/nerves-containers/docker_example.git
$ cd docker_example
$ MIX_TARGET=x86_64 mix deps.get
$ MIX_TARGET=x86_64 mix firmware
$ qemu-img create disk.img 20G
$ MIX_TARGET=x86_64 mix firmware.image disk.img
# or alternatively
# fwup -a -d disk.img -i _build/x86_64_dev/nerves/images/docker_example.fw  -t complete --max-size 42949672960
6

Possibly related commit Update fwup config to fill out data partition for qemu · nerves-containers/nerves_containers_x86_64@399bc4b · GitHub :smiley: - the repo is basically a fork of nerves_system_x86_64.

7

@steffend I added support for part of the issue that you ran into since it was trivially reproducible. I expect that this fixes the case when you create the disk.img file and do not use --max-size.

PR is at When writing regular files, check file size to support expansion by fhunleth · Pull Request #231 · fwup-home/fwup · GitHub. Manual build instructions for fwup are at fwup/docs/build_osx.md at main · fwup-home/fwup · GitHub, but no worries if you don’t have time to try it out.

The fact that --max-size doesn’t work is strange since there actually is a unit test that I thought exercised that case. I will investigate that next time I get a spare moment.

Thank you for bringing this to my attention.

3 Likes
8

I went through your test steps to verify that the final partition was properly expanded on your docker_example repository. Both using qmeu-img and the --max-size option work for me with the fwup PR modifications. I’m going to go ahead and make a fwup 1.11.0 release so that it’s generally available.

2 Likes