Problem

I recently ran into a situation where a production device was running out of disk space for a data intensive task. When inspecting, I noticed that the overall writeable space (/root) was much smaller than expected; 512MB vs 1.5GB

My particular device uses a Kingston 2 GB eMMC, so I generally expect the /root partition to be ~1.5GB of writable space. I made another image with busybox fdisk on to start poking and that the reported size is correct, but the ending sectors of the application partition were much smaller indicating it was not expanded as expected.

Some sidebar with Frank helped point to a possible cause ¬

This happens when fwup can’t detect the eMMC size or the user makes writes to an image file first and then uses dd or Balena Etcher to write the eMMC.

I’ve never seen fwup fail to detect, but it’s an OS call and the manufacturing machine in question is Windows, so maybe?

If someone replaced fwup with Etcher, that would do it, and seems like something that might be done out of frustration and not knowing what the real problem was.

So maybe a followup to fwup, but I haven’t reproduced manually and the crux of the issue is the sizing was not done correctly on first burn. So let’s resize…

Solution

Since I know the expected size of the eMMC, I just need to correctly size the partition table. In my case, it’s MBR and a 2 step process:

Adjust the partition table so the application partition is fully expanded
Resize the drive

I first tried fdisk, but could not get it to work. It may be a result of limited busybox version, but no matter what I tried it never changed the partition. So I decided to use fwup via the write_mbr() action in a custom task. This involved making a very small “firmware” with the new task, like we do with fwup-ops.conf, putting it on device and calling the new task:

Steps on host

Create mbr.fw utility file

Contents of `mbr.conf` used to make utility firmware

require-fwup-version="1.0.0"

##
# These variables are needed for the MBR defition. They are copied
# from the original fwup.conf
define(UBOOT_ENV_OFFSET, 2048)
define(UBOOT_ENV_COUNT, 256)  # 128 KB

define(BOOT_PART_OFFSET, 4096)
define(BOOT_PART_COUNT, 28672)

define(ROOTFS_A_PART_OFFSET, 32768)
define(ROOTFS_A_PART_COUNT, 262144)
define-eval(ROOTFS_B_PART_OFFSET, "${ROOTFS_A_PART_OFFSET} + ${ROOTFS_A_PART_COUNT}")
define(ROOTFS_B_PART_COUNT, ${ROOTFS_A_PART_COUNT})

define-eval(APP_PART_OFFSET, "${ROOTFS_B_PART_OFFSET} + ${ROOTFS_B_PART_COUNT}")
define(APP_PART_COUNT, 1048576)

# This MBR definition matches the original fwup.conf exactly
mbr mbr {
    partition 0 {
        block-offset = ${BOOT_PART_OFFSET}
        block-count = ${BOOT_PART_COUNT}
        type = 0xc # FAT32
        boot = true
    }
    partition 1 {
        block-offset = ${ROOTFS_A_PART_OFFSET}
        block-count = ${ROOTFS_A_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 2 {
        block-offset = ${ROOTFS_B_PART_OFFSET}
        block-count = ${ROOTFS_B_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 3 {
        block-offset = ${APP_PART_OFFSET}
        block-count = ${APP_PART_COUNT}
        type = 0x83 # Linux
        expand = true
    }
}

# To be called on device with ¬
#   fwup mbr.fw -t mbr_write -d /dev/mmcblk0 -U -q
task mbr_write {
    mbr_write(mbr)
}

fwup -c -f mbr.conf -o mbr.fw
scp mbr.fw <device-ip-address>:/tmp/mbr.fw

Look here if you do not have SSH access

Copying over SSH is generally easier. In my case, this was a remote production device so I manually copied the small binary via NervesHub remote console

File.read!("mbr.fw") |> IO.inspect(limit: :infinity)
# copied printed output

Then on NervesHub Remote Console

File.write!("/tmp/mbr.fw", <paste-clipboard>)

Steps on device

Rewrite partition map letting fwup expand it

cmd "fwup /tmp/mbr.fw -t mbr_write -d /dev/mmcblk0 -U -q"

Resize the newly mapped drive

iex()> cmd "umount /dev/mmcblk0p4"
0

iex()> cmd "resize.f2fs /dev/mmcblk0p4"

Info: MKFS version
  "Linux version 5.10.145 (buildroot@buildroot) (armv7-nerves-linux-gnueabihf-gcc (crosstool-NG 1.25.0.105_0389288) 12.2.0, GNU ld (crosstool-NG 1.25.0.105_0389288) 2.39) #1 PREEMPT Mon Aug 7 21:08:50 UTC 2023"
Info: FSCK version
  from "Linux version 5.10.145 (buildroot@buildroot) (armv7-nerves-linux-gnueabihf-gcc (crosstool-NG 1.25.0.105_0389288) 12.2.0, GNU ld (crosstool-NG 1.25.0.105_0389288) 2.39) #1 PREEMPT Mon Aug 7 21:08:50 UTC 2023"
    to "Linux version 5.10.145 (buildroot@buildroot) (armv7-nerves-linux-gnueabihf-gcc (crosstool-NG UNKNOWN) 13.2.0, GNU ld (crosstool-NG UNKNOWN) 2.40) #1 PREEMPT Fri Aug 2 17:18:06 UTC 2024"
Info: superblock features = 0 : 
Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
Info: Segments per section = 1
Info: Sections per zone = 1
Info: total FS sectors = 1048576 (512 MB)
Info: CKPT version = 4ab4168a
Info: Duplicate valid checkpoint to mirror position 1024 -> 512
Info: Write valid nat_bits in checkpoint
[FIX] (move_one_curseg_info:2921)  --> Move curseg[0] 3 -> 7 after 1600

[FIX] (move_one_curseg_info:2921)  --> Move curseg[1] 4 -> 8 after 1600

[FIX] (move_one_curseg_info:2921)  --> Move curseg[2] 3d -> 9 after 1600

[FIX] (move_one_curseg_info:2921)  --> Move curseg[3] 0 -> 5 after 1600

[FIX] (move_one_curseg_info:2921)  --> Move curseg[4] 1 -> 6 after 1600

[FIX] (move_one_curseg_info:2921)  --> Move curseg[5] 2 -> a after 1600

Info: Write valid nat_bits in checkpoint
Try to do defragement: Done
[migrate_ssa: 272] Info: Done to migrate SSA blocks: sum_blkaddr = 0xe00 -> 0x1200
[migrate_nat: 389] Info: Done to migrate NAT blocks: nat_blkaddr = 0xa00 -> 0xa00
[migrate_sit: 447] Info: Done to restore new SIT blocks: 0x600
[rebuild_checkpoint: 486] Info: Overprovision ratio = 4.220%
[rebuild_checkpoint: 487] Info: Overprovision segments = 32 (GC reserved = 30)
Info: Write valid nat_bits in checkpoint
[rebuild_checkpoint: 603] Info: Done to rebuild checkpoint blocks
[update_superblock: 765] Info: Done to update superblock

Done: 1.325540 secs
0