Better workflow for building custom systems

Nerves Forum Nerves Questions / Help
,
1

This is mostly based on a Slack discussion, but I believe it’s better suited here.

I recently tried to build my first “custom” nerves system from my Mac, and while I appreciate the effort it has gone into making it a smooth experience, it still has a lot of warts.

In my experience it’s mostly about Buildroot, and not particularly about nerves.

The suggestion I received on Slack is that Docker for Mac is not very well suited for this purpose, since you can’t easily access the docker filesystem or cache the build artefacts etc etc. I therefore moved to an Ubuntu VM.

@fhunleth said on Slack:

I’m not sure how many others do this, but I avoid building systems with mix. My workflow is to checkout both nerves_system_br and the custom system, run create_build.sh to create a build directory and then run make in the build directory. To use the custom system, I source the ./nerves-env.sh from the build directory. It sets the NERVES_SYSTEM and NERVES_TOOLCHAIN environment variables which when you go to build your Elixir project will cause it to use your custom system instead.
I have a couple utility scripts to automate this slightly since I work with about 10 custom systems and sometimes I like to build them all.
This setup also allows me to have a global download directory to avoid redownloading things.

I hope others knowledgable with nerves will contribute to this post so that it can serve as interim documentation!

2 Likes
2

Paging @fhunleth - you mention having a global download directory, can you please elaborate?

Each custom system (rpi3, for example) takes 4-5GB of hard drive space, depending on the selected packages. Is there a way to share downloads, avoiding cloning the entire raspberry-pi-linux repository on every make etc?

3

The create_build.sh script sets up the Buildroot download cache directory to be ~/.nerves/cache/buildroot by default. Since I do a lot of non-Nerves Buildroot work as well, I symlink that directory to my global Buildroot cache but you don’t need to do that.

As a test, you can run make source in the output directory created by create_build.sh. That will download everything so that future builds can be done offline. Hopefully, you’ll see the cache get populated.

4

So, I successfully built a custom system, sourced nerves-env.sh from the build directory. I can check that $NERVES_SYSTEM and $NERVES_TOOLCHAIN are set correctly in my bash session.

However, when I go into my app (taken directly from the nerves-examples, e.g. hello-phoenix) and do mix firmware I get:

Env
  MIX_TARGET:   host
  MIX_ENV:      dev

NERVES_SYSTEM is unset
NERVES_TOOLCHAIN is unset

[...]

Should I also set $MIX_TARGET ? Why isn’t mix picking up my environment? I’m confused!

5

Yes, you still need to set MIX_TARGET=rpi3 or however you refer to the target in your project’s mix.exs file. I sometimes modify the mix.exs to make my preferred target the default instead of host.

6

I ask:

if I build a custom system in a VM, what are the files I need to copy to my main machine so that I can continue developing there? Doing the SD card dance in virtualbox is getting old…

@jschneck Says:

You can call make system from the output dir of your custom system. This will construct a tar of the artifact which you can copy over to your host.
Then you can reference this by pointing the env var NERVES_SYSTEM to its decompressed location.

MIX_TARGET should be set to some tag that allows mix to satisfy the dependency in the mix deps, IE it adds the location to the custom systems mix project. It needs this still, because it needs the info located inside the custom system deps nerves.exs package config

7

In the time since this question was originally asked, the state of support for custom Nerves systems on OSX has improved. With projects that depend on nerves ~> 0.7.0 and using the latest nerves_bootstrap archive, you can configure your custom system using the new nerves.system.shell task, either from your project’s source directory or from the custom system’s source directory. This should work approximately the same on Linux and OSX (and maybe Windows if you have WSL installed, support is iffy on this still).

This allows you to configure your project to depend on your custom system (e.g. using a path: dependency) and have mix firmware build it for you whenever the relevant source files change. You can also use the mix nerves.system.shell task to access a Docker container on OSX (or a sub-shell on Linux) that will allow you to configure your system (e.g. using make menuconfig.

As always, you’ll still need to set MIX_TARGET appropriately so that your mix.exs file will know which system dependency you want to use.

1 Like