Rpi0 squashfs error, A/B partition switch not working?

Nerves Forum
,
1

I’ve just run into a crash after a local fwup on a rpi0 nerves device, and it presents the following boot loop:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.4.51 (buildroot@buildroot) (gcc version 10.2.0 (crosstool-NG 1.24.0.299_6729a76)) #
1 PREEMPT Sat Sep 4 09:50:00 UTC 2021
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi Zero W Rev 1.1
[    0.000000] Memory policy: Data cache writeback
[    0.000000] Reserved memory: created CMA memory pool at 0x0fc00000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 81200
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.
enable_hdmi=1 snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 s
msc95xx.macaddr=B8:27:EB:21:70:47 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  root=/dev/mmcblk0p2 rootw
ait modules-load=dwc2,g_ether g_ether.host_addr=00:22:82:ff:ff:20 console=ttyAMA0,115200
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 250076K/327680K available (5535K kernel code, 202K rwdata, 1088K rodata, 176K init, 326K bs
s, 12068K reserved, 65536K cma-reserved)
[    0.000000] random: get_random_u32 called from cache_random_seq_create+0x80/0x130 with crng_init=0
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000025] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.000060] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.000151] bcm2835: system timer (irq = 27)
[    0.000590] Console: colour dummy device 80x30
[    0.000658] Calibrating delay loop... 697.95 BogoMIPS (lpj=3489792)
[    0.060273] pid_max: default: 32768 minimum: 301
[    0.060763] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.060787] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.062206] Disabling memory control group subsystem
[    0.062316] CPU: Testing write buffer coherency: ok
[    0.063575] Setting up static identity map for 0x8200 - 0x8238
[    0.063782] rcu: Hierarchical SRCU implementation.
[    0.064411] devtmpfs: initialized
[    0.073816] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[    0.074347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.074382] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.081868] pinctrl core: initialized pinctrl subsystem
[    0.082993] NET: Registered protocol family 16
[    0.085620] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[    0.092143] Serial: AMBA PL011 UART driver
[    0.092797] ramoops: error in header, 4
[    0.100573] ramoops: error in header, 2
[    0.123912] printk: console [pstore-1] enabled
[    0.124319] pstore: Registered ramoops as persistent store backend
[    0.124430] ramoops: using 0x100000@0xb000000, ecc: 16
[    0.126608] bcm2835-mbox 2000b880.mailbox: mailbox enabled
[    0.150731] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-09-02 21:16, variant start_x
[    0.160746] raspberrypi-firmware soc:firmware: Firmware hash is 4439d2aaa6c376a2d1ef4402f142e1cf4de37c43
[    0.186695] bcm2835-dma 20007000.dma: DMA legacy API manager, dmachans=0x1
[    0.189063] SCSI subsystem initialized
[    0.189494] usbcore: registered new interface driver usbfs
[    0.189754] usbcore: registered new interface driver hub
[    0.190033] usbcore: registered new device driver usb
[    0.192131] clocksource: Switched to clocksource timer
[    0.207964] thermal_sys: Registered thermal governor 'step_wise'
[    0.208575] NET: Registered protocol family 2
[    0.209812] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.210039] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.210206] TCP bind hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.210441] TCP: Hash tables configured (established 4096 bind 4096)
[    0.210668] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.210868] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.211356] NET: Registered protocol family 1
[    0.212725] Initialise system trusted keyrings
[    0.213093] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[    0.227292] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.228288] Key type asymmetric registered
[    0.228403] Asymmetric key parser 'x509' registered
[    0.228564] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    0.231178] bcm2708_fb soc:fb: FB found 1 display(s)
[    0.241277] Console: switching to colour frame buffer device 82x26
[    0.245953] bcm2708_fb soc:fb: Registered framebuffer for display 0, size 656x416
[    0.249475] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.251743] bcm2835-rng 20104000.rng: hwrng registered
[    0.252487] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[    0.253514] vc-sm: Videocore shared memory driver
[    0.254261] gpiomem-bcm2835 20200000.gpiomem: Initialised: Registers at 0x20200000
[    0.269296] brd: module loaded
[    0.276618] loop: module loaded
[    0.278125] spi-bcm2835 20204000.spi: could not get clk: -517
[    0.278889] libphy: Fixed MDIO Bus: probed
[    0.279077] PPP generic driver version 2.4.2
[    0.279497] PPP BSD Compression module registered
[    0.279678] PPP Deflate Compression module registered
[    0.279785] PPP MPPE Compression module registered
[    0.279884] NET: Registered protocol family 24
[    0.280186] usbcore: registered new interface driver asix
[    0.280359] usbcore: registered new interface driver ax88179_178a
[    0.280591] usbcore: registered new interface driver cdc_ether
[    0.280747] usbcore: registered new interface driver net1080
[    0.280976] usbcore: registered new interface driver cdc_subset
[    0.281152] usbcore: registered new interface driver zaurus
[    0.281425] usbcore: registered new interface driver cdc_ncm
[    0.281592] usbcore: registered new interface driver huawei_cdc_ncm
[    0.281831] usbcore: registered new interface driver cdc_mbim
[    0.282028] usbcore: registered new interface driver cdc_acm
[    0.282364] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    0.282627] usbcore: registered new interface driver cdc_wdm
[    0.282904] usbcore: registered new interface driver usbserial_generic
[    0.283131] usbserial: USB Serial support registered for generic
[    0.283287] usbcore: registered new interface driver cypress_m8
[    0.283508] usbserial: USB Serial support registered for DeLorme Earthmate USB
[    0.283651] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[    0.283869] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[    0.284098] usbcore: registered new interface driver ftdi_sio
[    0.284243] usbserial: USB Serial support registered for FTDI USB Serial Device
[    0.284489] usbcore: registered new interface driver option
[    0.284638] usbserial: USB Serial support registered for GSM modem (1-port)
[    0.284961] udc-core: couldn't find an available UDC - added [g_ether] to list of pending drivers
[    0.285145] i2c /dev entries driver
[    0.286236] Driver for 1-wire Dallas network protocol.
[    0.287727] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[    0.288178] bcm2835-cpufreq: min=700000 max=1000000
[    0.289366] sdhci: Secure Digital Host Controller Interface driver
[    0.289485] sdhci: Copyright(c) Pierre Ossman
[    0.290118] mmc-bcm2835 20300000.mmcnr: could not get clk, deferring probe
[    0.290805] sdhost-bcm2835 20202000.mmc: could not get clk, deferring probe
[    0.291194] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.291871] ledtrig-cpu: registered to indicate activity on CPUs
[    0.293507] vchiq: vchiq_init_state: slot_zero = (ptrval)
[    0.295160] [vc_sm_connected_init]: start
[    0.303256] [vc_sm_connected_init]: end - returning 0
[    0.307272] NET: Registered protocol family 10
[    0.308870] Segment Routing with IPv6
[    0.309266] NET: Registered protocol family 17
[    0.309854] Loading compiled-in X.509 certificates
[    0.310514] pstore: Using crash dump compression: deflate
[    0.313327] uart-pl011 20201000.serial: cts_event_workaround enabled
[    0.313620] 20201000.serial: ttyAMA0 at MMIO 0x20201000 (irq = 81, base_baud = 0) is a PL011 rev2
[    1.127086] printk: console [ttyAMA0] enabled
[    1.133597] 20215040.serial: ttyS0 at MMIO 0x0 (irq = 53, base_baud = 50000000) is a 16550
[    1.142811] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[    1.156310] mmc-bcm2835 20300000.mmcnr: mmc_debug:0 mmc_debug2:0
[    1.162661] mmc-bcm2835 20300000.mmcnr: DMA channel allocated
[    1.197176] sdhost: log_buf @ (ptrval) (4fd10000)
[    1.218202] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[    1.233783] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.240975] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.249587] random: fast init done
[    1.253157] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    1.259812] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[    1.265675] of_cfs_init
[    1.268437] of_cfs_init: OK
[    1.292816] uart-pl011 20201000.serial: no DMA platform data
[    1.298745] Waiting for root device /dev/mmcblk0p2...
[    1.369380] mmc0: host does not support reading read-only switch, assuming write-enable
[    1.383301] mmc0: new high speed SDHC card at address aaaa
[    1.390267] mmcblk0: mmc0:aaaa SC32G 29.7 GiB
[    1.399624]  mmcblk0: p1 p2 p3
[    1.418633] mmc1: new high speed SDIO card at address 0001
[    1.432868] VFS: Mounted root (squashfs filesystem) readonly on device 179:2.
[    1.444550] devtmpfs: mounted
[    1.448195] Freeing unused kernel memory: 176K
[    1.452919] This architecture does not have kernel memory protection.
[    1.459536] Run /sbin/init as init process
[    1.483662] SQUASHFS error: zlib decompression failed, data probably corrupt
[    1.490863] SQUASHFS error: squashfs_read_data failed to read block 0x2791083
[    1.498437] SQUASHFS error: zlib decompression failed, data probably corrupt
[    1.505668] SQUASHFS error: squashfs_read_data failed to read block 0x2791083
[    1.513051] Starting init: /sbin/init exists but couldn't execute it (error -5)
[    1.520430] Run /etc/init as init process
[    1.528144] Run /bin/init as init process
[    1.532522] Run /bin/sh as init process
[    1.549462] SQUASHFS error: zlib decompression failed, data probably corrupt
[    1.556854] SQUASHFS error: squashfs_read_data failed to read block 0x2791083
[    1.564318] Starting init: /bin/sh exists but couldn't execute it (error -5)
[    1.571448] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux D
ocumentation/admin-guide/init.rst for guidance.
[    1.585791] CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.51 #1
[    1.591593] Hardware name: BCM2835
[    1.595083] [<80014da8>] (unwind_backtrace) from [<80012778>] (show_stack+0x10/0x14)
[    1.602951] [<80012778>] (show_stack) from [<8001e018>] (panic+0xfc/0x2f4)
[    1.609949] [<8001e018>] (panic) from [<8056902c>] (kernel_init+0xbc/0xec)
[    1.616893] [<8056902c>] (kernel_init) from [<800090d8>] (ret_from_fork+0x14/0x3c)
[    1.624576] Exception stack(0x8f841fb0 to 0x8f841ff8)
[    1.629688] 1fa0:                                     00000000 00000000 00000000 00000000
[    1.637983] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.646275] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    1.661385] Rebooting in 10 seconds..

It looks like it might be a corrupted SD card block, but I would have expected nerves to revert to the other boot partition in such cases, is there anything that can be done? This would suck if it happened to a unit in the field when remotely updating.

2

By default, Nerves does not automatically revert if you have a bad firmware update. Nerves gives you tools to do this for platforms using U-Boot like the Beaglebone. It looks possible to implement this on the Raspberry Pis. I don’t know anyone who has done it yet.

As for why this error occurred, it does seem like it might be a problem with the MicroSD card. I have not seen it.

1 Like
3

I see, what does Nerves use on the Raspberry Pis?

4

Nerves uses the Raspberry Pi’s bootloader. It’s closed source, so I haven’t found one definitive place to go for all of its features. Here are some references:

There are two other ways to support automatically failing back on the Raspberry Pi. They have issues which is one of the reasons why they’re not in Nerves. They might be ok for you:

  1. Include U-Boot. It’s possible to have the Raspberry Pi’s bootloader load U-Boot and then have U-Boot load Linux, etc. just like boards running U-Boot. We decided against this option just because it felt like too many pieces and would be hard for us to support.
  2. Use an initramfs that has the logic to automatically fail back. I made an attempt at this at GitHub - nerves-project/nerves_initramfs: An initramfs for early boot handling of Nerves devices (Work in progress!). A while back, @jjcarstens made a branch of the Nerves Raspberry Pi systems that used it. It “works”, but has a couple disadvantages. First, it’s not possible to revert if the Linux kernel or initramfs are messed up. Since these are updated on every software update, this strategy has a serious flaw even though it probably would handle 90+% of the failed updates that I encounter. The other problem is that the initramfs program got too complicated and I no longer feel good about it being reliable. This could be fixed, but once I learned about tryboot, I felt like investigating that would be a better use of time.
2 Likes