ivanhercaz

ivanhercaz

Check and include files in "/data" directory when Nerves is initialized

Hi everyone!
I am working with a customized image of Nerves for Raspberry Pi 4 because I need to include some Buildroot packages to interact with a Python application that I include in the priv directory of my Nerves app. I know it is not recommend to use priv directory, but it was the fastest option to test it.

This Python application has two main components: the app and a configuration file. I need to put the configuration file on a writable path, and the one I have seen recommended is /data, but I need to include this file before Nerves start the application supervisor.

Is there any way in which I can specify that, for example, data_app directory will be included in /data when I make the firmware?

The Python application must be started when Nerves start, so I apply the next workaround:

  • Nerves started.
  • Check if the configuration file exists on /data, if not, copy it from /priv/conf.file to /data/conf.file`.
  • Start the supervisor with its children, where one of them start the Python application that need the bridge.

Now, at least I can replace the configuration file using sftp. But I am still interested in what I ask above, because I think it would be easier and even the Python application may be in /data.

Thanks in advance!

First 5 of 5 Posts! Switch mode

LostKobrakai

LostKobrakai

ivanhercaz

ivanhercaz

Wuou! Thank you very much for linking to that section of the documentation! I read and read, but I overlooked that part.

ivanhercaz

ivanhercaz

Hi @LostKobrakai!
May be you or someone else could help me with the problem I am having trying to insert this extra files inside of a read and write partition.

My first try was to create a rootfs_overlay/data/ and insert in this directory the Python application and some other configuration files. However, I get:

|nerves| Building OTP Release…

  • skipping runtime configuration (config/runtime.exs not found)
  • creating _build/custom_rpi4_dev/rel/nerves_app/releases/0.1.0/vm.args
    ** (Mix) The firmware contains overlay files which reference directories that are
    mounted as file systems on the device. The filesystem mount will completely
    overwrite the overlay and these files will be lost.

Remove the following overlay directories and build the firmware again:

  • rootfs_overlay/data

Due to this message I understand I can’t do it, thus I tried to configure a new partition following the instructions of the section Partitions from the documentation.

First, I copy the fwup.conf file from my customized Nerves system for Raspberry Pi 4 to config/fwup.conf, because my find deps -name fwup.conf (mentioned in the documentation) doesn’t returns this file from the deps directory. Then I specify this change in config/config.exs:

config :nerves, :firmware,
  rootfs_overlay: "rootfs_overlay",
  fwup_conf: "config/fwup.conf"

In fwup.conf I don’t touch the application partition (APP_PART, /data), but add EXTRA_PART, and the result of my addition is the next fragment:

define(UBOOT_ENV_OFFSET, 16)
define(UBOOT_ENV_COUNT, 16)  # 8 KB

define(BOOT_A_PART_OFFSET, 63)
define(BOOT_A_PART_COUNT, 38630)
define-eval(BOOT_B_PART_OFFSET, "${BOOT_A_PART_OFFSET} + ${BOOT_A_PART_COUNT}")
define(BOOT_B_PART_COUNT, ${BOOT_A_PART_COUNT})

# Let the rootfs have room to grow up to 128 MiB and align it to the nearest 1
# MB boundary
define(ROOTFS_A_PART_OFFSET, 77324)
define(ROOTFS_A_PART_COUNT, 289044)
define-eval(ROOTFS_B_PART_OFFSET, "${ROOTFS_A_PART_OFFSET} + ${ROOTFS_A_PART_COUNT}")
define(ROOTFS_B_PART_COUNT, ${ROOTFS_A_PART_COUNT})

# Application partition. This partition can occupy all of the remaining space.
# Size it to fit the destination.
define-eval(APP_PART_OFFSET, "${ROOTFS_B_PART_OFFSET} + ${ROOTFS_B_PART_COUNT}")
define(APP_PART_COUNT, 1048576)

# Resources Extra data partition.
# Size it to fit the destination.
define-eval(EXTRA_PART_OFFSET, "${APP_PART_OFFSET} + ${APP_PART_COUNT}")
define(EXTRA_PART_COUNT, 1048576)

In the mapping partition (mbr) I add an extra partition (partition 3), removing the expand = true from partition 2:

mbr mbr-a {
    partition 0 {
        block-offset = ${BOOT_A_PART_OFFSET}
        block-count = ${BOOT_A_PART_COUNT}
        type = 0xc # FAT32
        boot = true
    }
    partition 1 {
        block-offset = ${ROOTFS_A_PART_OFFSET}
        block-count = ${ROOTFS_A_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 2 {
        block-offset = ${APP_PART_OFFSET}
        block-count = ${APP_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 3 {
        block-offset = ${EXTRA_PART_OFFSET}
        block-count = ${EXTRA_PART_COUNT}
        type = 0x83 # Linux
    }
}

I shared the mbr-a, but it is the same addition in mbr-b, without touching the rest of partitions with the exception of the expand = true of partition 2.

And, finally I run unsquashfs ~/.nerves/artifacts/<cached_system_name>/images/rootfs.squashfs of my system and copy erlinit.config file from squasfs-root/etc/ to rootfs_overlay/etc/erlinit.config and, in the part to mount the partitions, I add one line for the extra partition (line 3):

-m /dev/mmcblk0p1:/boot:vfat:ro,nodev,noexec,nosuid:
-m /dev/mmcblk0p3:/root:f2fs:nodev:
-m /dev/mmcblk0p4:/extra:ext4:nodev:
-m pstore:/sys/fs/pstore:pstore:nodev,noexec,nosuid:
-m tmpfs:/sys/fs/cgroup:tmpfs:nodev,noexec,nosuid:mode=755,size=1024k
-m cpu:/sys/fs/cgroup/cpu:cgroup:nodev,noexec,nosuid:cpu

Before to run mix firmware I create rootfs_overlay/extra directory and insert an empty test file (test.txt).

After run mix firmware I run mix burn to burn the SD and when the device is up, I entered in its iex console with SSH and check for erlinit log:

iex(1)> RingLogger.grep ~r/erlinit/
00:00:04.071 [warn]  erlinit: Cannot mount /dev/mmcblk0p3 at /root: Invalid argument
00:00:04.071 [warn]  erlinit: Cannot mount /dev/mmcblk0p4 at /extra: No such device
:ok

I decided to test myself:

iex(2)> File.touch("/root/test.txt")
:ok
iex(3)> File.touch("/extra/test.txt")
{:error, :erofs}
iex(4)> Nerves.Runtime.reboot
:ok

Then I reboot to check if I get the same warning for /dev/mmcblk0p3:root:

iex(1)> RingLogger.grep ~r/erlinit/  
00:00:04.092 [warn]  erlinit: Cannot mount /dev/mmcblk0p4 at /extra: No such device
:ok
iex(2)> File.touch("/extra/test.txt")
{:error, :erofs}

And effectively, I don’t get it, but I still get the "no such device for /dev/mmcblk0p4:/extra. If I dive in the /dev directory I can see I have the next devices:

iex(5)> cmd "find /dev -name mmcblk0p*"
/dev/mmcblk0p4
/dev/mmcblk0p3
/dev/mmcblk0p2
/dev/mmcblk0p1

Any idea about this situation?

Thanks in advance for your effort to read this long post.

ivanhercaz

ivanhercaz

Before to begin all this process I described, that it isn’t work, I tried what @fhunleth said in the post How to deploy file resource on Nerves device? - #2 by fhunleth

However, if I touch the file or try to unzip it (it is a zip file), it returns me an error that notice is read-only:

iex> File.touch(file_zipped)
{:error, :erofs}

If then I use sftp to replace the file with the same one, it “seems” isn’t read-only more, so if I use File.touch/1 I get an :ok, not an error. But if try to unzip it I still gets the {:error, :erofs}:

iex>{:ok, file} = File.read(filepath_in_data)
iex> :zip.unzip(file)                                                         
{:error,
 {:EXIT,
  {{:badmatch, {:error, :erofs}},
  ...
}}}

In addition, I want to avoid to run sftp for this case.

ivanhercaz

ivanhercaz

Interesting, the below code works even when it is copied from the priv directory:

iex> cmd "unzip /data/#{filename} -d /data/"

Or, as I use it in my code:

{_result, 0} = MuonTrap.cmd("unzip", ["-o", file, "-d", "/data"])

So at least this workaround helps me while I think in the errors of the above posts.

Where Next?

Trending in Questions Top

jonnycharles
I’m in search of an Elixir library that offers PDF generation capabilities similar to Ruby’s Prawn. While there have been discussions abo...
New
spammy
I’m looking to build a personal workflow to quickly deploy web applications written in elixir/phoenix, for local consumption (ie not on t...
New
silverdr
Using Phoenix.LiveView.TagEngine as an EEx.Engine is deprecated! To compile HEEx, use Phoenix.LiveView.TagEngine.compile/2 instead. Sta...
New
saveman71
Hello ! We want new/edit form pages to POST/PUT to their own URL rather than the resources REST defaults (post /things, put /things/:id)...
New
dli
Before I dive in myself, did anyone successfully sprinkle Hologram into their existing LiveView app? Looking for hints regarding: Addi...
New
bottlenecked
Hi all, I wanted to ask how the community is dealing with post-release steps. Today we have Ecto migrations, which make sure that the db...
New
michallepicki
I am using Oban and occasionally, shortly after a deployment, a handful of jobs can fail because of dependency on other parts of the syst...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge &amp; Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New

We're in Beta

About us Mission Statement