Check and include files in "/data" directory when Nerves is initialized

Hi everyone!
I am working with a customized image of Nerves for Raspberry Pi 4 because I need to include some Buildroot packages to interact with a Python application that I include in the priv directory of my Nerves app. I know it is not recommend to use priv directory, but it was the fastest option to test it.

This Python application has two main components: the app and a configuration file. I need to put the configuration file on a writable path, and the one I have seen recommended is /data, but I need to include this file before Nerves start the application supervisor.

Is there any way in which I can specify that, for example, data_app directory will be included in /data when I make the firmware?

The Python application must be started when Nerves start, so I apply the next workaround:

  • Nerves started.
  • Check if the configuration file exists on /data, if not, copy it from /priv/conf.file to /data/conf.file`.
  • Start the supervisor with its children, where one of them start the Python application that need the bridge.

Now, at least I can replace the configuration file using sftp. But I am still interested in what I ask above, because I think it would be easier and even the Python application may be in /data.

Thanks in advance!


Wuou! Thank you very much for linking to that section of the documentation! I read and read, but I overlooked that part.

Hi @LostKobrakai!
May be you or someone else could help me with the problem I am having trying to insert this extra files inside of a read and write partition.

My first try was to create a rootfs_overlay/data/ and insert in this directory the Python application and some other configuration files. However, I get:

|nerves| Building OTP Release…

  • skipping runtime configuration (config/runtime.exs not found)
  • creating _build/custom_rpi4_dev/rel/nerves_app/releases/0.1.0/vm.args
    ** (Mix) The firmware contains overlay files which reference directories that are
    mounted as file systems on the device. The filesystem mount will completely
    overwrite the overlay and these files will be lost.

Remove the following overlay directories and build the firmware again:

  • rootfs_overlay/data

Due to this message I understand I can’t do it, thus I tried to configure a new partition following the instructions of the section Partitions from the documentation.

First, I copy the fwup.conf file from my customized Nerves system for Raspberry Pi 4 to config/fwup.conf, because my find deps -name fwup.conf (mentioned in the documentation) doesn’t returns this file from the deps directory. Then I specify this change in config/config.exs:

config :nerves, :firmware,
  rootfs_overlay: "rootfs_overlay",
  fwup_conf: "config/fwup.conf"

In fwup.conf I don’t touch the application partition (APP_PART, /data), but add EXTRA_PART, and the result of my addition is the next fragment:

define(UBOOT_ENV_OFFSET, 16)
define(UBOOT_ENV_COUNT, 16)  # 8 KB

define(BOOT_A_PART_OFFSET, 63)
define(BOOT_A_PART_COUNT, 38630)

# Let the rootfs have room to grow up to 128 MiB and align it to the nearest 1
# MB boundary
define(ROOTFS_A_PART_OFFSET, 77324)
define(ROOTFS_A_PART_COUNT, 289044)

# Application partition. This partition can occupy all of the remaining space.
# Size it to fit the destination.
define(APP_PART_COUNT, 1048576)

# Resources Extra data partition.
# Size it to fit the destination.
define(EXTRA_PART_COUNT, 1048576)

In the mapping partition (mbr) I add an extra partition (partition 3), removing the expand = true from partition 2:

mbr mbr-a {
    partition 0 {
        block-offset = ${BOOT_A_PART_OFFSET}
        block-count = ${BOOT_A_PART_COUNT}
        type = 0xc # FAT32
        boot = true
    partition 1 {
        block-offset = ${ROOTFS_A_PART_OFFSET}
        block-count = ${ROOTFS_A_PART_COUNT}
        type = 0x83 # Linux
    partition 2 {
        block-offset = ${APP_PART_OFFSET}
        block-count = ${APP_PART_COUNT}
        type = 0x83 # Linux
    partition 3 {
        block-offset = ${EXTRA_PART_OFFSET}
        block-count = ${EXTRA_PART_COUNT}
        type = 0x83 # Linux

I shared the mbr-a, but it is the same addition in mbr-b, without touching the rest of partitions with the exception of the expand = true of partition 2.

And, finally I run unsquashfs ~/.nerves/artifacts/<cached_system_name>/images/rootfs.squashfs of my system and copy erlinit.config file from squasfs-root/etc/ to rootfs_overlay/etc/erlinit.config and, in the part to mount the partitions, I add one line for the extra partition (line 3):

-m /dev/mmcblk0p1:/boot:vfat:ro,nodev,noexec,nosuid:
-m /dev/mmcblk0p3:/root:f2fs:nodev:
-m /dev/mmcblk0p4:/extra:ext4:nodev:
-m pstore:/sys/fs/pstore:pstore:nodev,noexec,nosuid:
-m tmpfs:/sys/fs/cgroup:tmpfs:nodev,noexec,nosuid:mode=755,size=1024k
-m cpu:/sys/fs/cgroup/cpu:cgroup:nodev,noexec,nosuid:cpu

Before to run mix firmware I create rootfs_overlay/extra directory and insert an empty test file (test.txt).

After run mix firmware I run mix burn to burn the SD and when the device is up, I entered in its iex console with SSH and check for erlinit log:

iex(1)> RingLogger.grep ~r/erlinit/
00:00:04.071 [warn]  erlinit: Cannot mount /dev/mmcblk0p3 at /root: Invalid argument
00:00:04.071 [warn]  erlinit: Cannot mount /dev/mmcblk0p4 at /extra: No such device

I decided to test myself:

iex(2)> File.touch("/root/test.txt")
iex(3)> File.touch("/extra/test.txt")
{:error, :erofs}
iex(4)> Nerves.Runtime.reboot

Then I reboot to check if I get the same warning for /dev/mmcblk0p3:root:

iex(1)> RingLogger.grep ~r/erlinit/  
00:00:04.092 [warn]  erlinit: Cannot mount /dev/mmcblk0p4 at /extra: No such device
iex(2)> File.touch("/extra/test.txt")
{:error, :erofs}

And effectively, I don’t get it, but I still get the "no such device for /dev/mmcblk0p4:/extra. If I dive in the /dev directory I can see I have the next devices:

iex(5)> cmd "find /dev -name mmcblk0p*"

Any idea about this situation?

Thanks in advance for your effort to read this long post.

Before to begin all this process I described, that it isn’t work, I tried what @fhunleth said in the post How to deploy file resource on Nerves device? - #2 by fhunleth

However, if I touch the file or try to unzip it (it is a zip file), it returns me an error that notice is read-only:

iex> File.touch(file_zipped)
{:error, :erofs}

If then I use sftp to replace the file with the same one, it “seems” isn’t read-only more, so if I use File.touch/1 I get an :ok, not an error. But if try to unzip it I still gets the {:error, :erofs}:

iex>{:ok, file} =
iex> :zip.unzip(file)                                                         
  {{:badmatch, {:error, :erofs}},

In addition, I want to avoid to run sftp for this case.

Interesting, the below code works even when it is copied from the priv directory:

iex> cmd "unzip /data/#{filename} -d /data/"

Or, as I use it in my code:

{_result, 0} = MuonTrap.cmd("unzip", ["-o", file, "-d", "/data"])

So at least this workaround helps me while I think in the errors of the above posts.