jshprentz

jshprentz

Getting Started with Libcamera in Nerves 1.11

After I upgraded my Raspberry Pi Zero W to Nerves v1.11.0 (and more recently to v1.11.1), my application could no longer access the Raspberry Pi Camera Module 2 via the Picam library. The nerves_system_rpi0 v1.28.0 changelog explains:

Switch from the Raspberry Pi’s deprecated MMAL media support to DRM and libcamera. This is a big change if you use the display or camera that has been a long time coming. Please plan some time to make the upgrade.

The libcamera documentation provided a starting point for exploring new Nerves camera interface. I tried to list available cameras:

iex(1)> cmd("rpicam-jpeg --list-cameras")
No cameras available!
0

Two weeks of intermittent experimentation yielded these observations:

  1. The libcamera apps were included in the Nerves firmware. (Good)
  2. Over a dozen /dev/video** devices were created at boot time. (Good)
  3. The firmware lacked an overlay for the camera. (Bad)

As described in the libcamera configuration documentation,
Raspberry Pi OS automatically detects most common cameras and loads an appropriate overlay; Nerves does not. This may be a Nerves feature (reduce bloat) or a Nerves bug (something is missing).

Pull request #221 for nerves_system_rpi4 provided an example of adding a camera overlay to fwup.conf. More generally, the Nerves documentation includes the sections Overwriting Files in the Boot Partition and Device Tree Overlays.

Copy fwup.conf and config.txt to the project’s config directory for customization.

joel@Joels-Dell-XPS:~/hello_nerves$ cp deps/nerves_system_rpi0/fwup.conf config/
joel@Joels-Dell-XPS:~/hello_nerves$ cp deps/nerves_system_rpi0/config.txt config/

Modify config/config.exs to use the custom fwup.conf.

*** config/config.exs.orig      2024-08-07 11:00:34.563016026 -0400
--- config/config.exs   2024-08-08 11:52:27.443095965 -0400
***************
*** 12,16 ****
  # https://hexdocs.pm/nerves/advanced-configuration.html for details.

! config :nerves, :firmware, rootfs_overlay: "rootfs_overlay"

  # Set the SOURCE_DATE_EPOCH date for reproducible builds.
--- 12,16 ----
  # https://hexdocs.pm/nerves/advanced-configuration.html for details.

! config :nerves, :firmware, rootfs_overlay: "rootfs_overlay", fwup_conf: "config/fwup.conf"

  # Set the SOURCE_DATE_EPOCH date for reproducible builds.

Modify config/fwup.conf to

  1. Use the custom config.txt.
  2. Add a file resource for the IMX219 camera overlay.
  3. Add steps in three tasks to copy the overlay file to the firmware boot partition. The first two tasks write to the A partion; the last task writes to the B partition.
*** deps/nerves_system_rpi0/fwup.conf   2024-08-07 23:30:25.000000000 -0400
--- config/fwup.conf    2024-08-08 12:34:46.493093918 -0400
***************
*** 19,21 ****
  file-resource config.txt {
!     host-path = "${NERVES_SYSTEM}/images/config.txt"
  }
--- 19,21 ----
  file-resource config.txt {
!     host-path = "${NERVES_APP}/config/config.txt"
  }
***************
*** 56,57 ****
--- 56,61 ----

+ file-resource imx219.dtbo {
+     host-path = "${NERVES_SYSTEM}/images/rpi-firmware/overlays/imx219.dtbo"
+ }
+
  mbr mbr-a {
***************
*** 150,151 ****
--- 154,156 ----
      on-resource ramoops.dtbo { fat_write(${BOOT_A_PART_OFFSET}, "overlays/ramoops.dtbo") }
+     on-resource imx219.dtbo { fat_write(${BOOT_A_PART_OFFSET}, "overlays/imx219.dtbo") }

***************
*** 215,216 ****
--- 220,222 ----
      on-resource ramoops.dtbo { fat_write(${BOOT_A_PART_OFFSET}, "overlays/ramoops.dtbo") }
+     on-resource imx219.dtbo { fat_write(${BOOT_A_PART_OFFSET}, "overlays/imx219.dtbo") }
      on-resource rootfs.img {
***************
*** 287,288 ****
--- 293,295 ----
      on-resource ramoops.dtbo { fat_write(${BOOT_B_PART_OFFSET}, "overlays/ramoops.dtbo") }
+     on-resource imx219.dtbo { fat_write(${BOOT_B_PART_OFFSET}, "overlays/imx219.dtbo") }
      on-resource rootfs.img {

Modify config/config.txt to load the camera overlay.

*** deps/nerves_system_rpi0/config.txt  2024-08-07 23:30:25.000000000 -0400
--- config/config.txt   2024-08-09 11:11:53.793018875 -0400
***************
*** 55 ****
--- 55,59 ----
  #dtparam=act_led_trigger=none
+
+ # Load the Raspberry Pi Camera 2 overlay
+ dtoverlay=imx219
+ camera_auto_detect=0

Rebuild the firmware.

joel@Joels-Dell-XPS:~/hello_nerves$ mix firmware
==> nerves
==> hello_nerves

Nerves environment
  MIX_TARGET:   rpi0
  MIX_ENV:      dev

Generated hello_nerves app
|nerves| Building OTP Release...

* [Nerves] validating vm.args
* skipping runtime configuration (config/runtime.exs not found)
* creating _build/rpi0_dev/rel/hello_nerves/releases/0.1.0/vm.args
Updating base firmware image with Erlang release...
Copying rootfs_overlay: /home/joel/hello_nerves/_build/rpi0_dev/nerves/rootfs_overlay
Copying rootfs_overlay: /home/joel/hello_nerves/rootfs_overlay
Building /home/joel/hello_nerves/_build/rpi0_dev/nerves/images/hello_nerves.fw...
Firmware built successfully! πŸŽ‰

Now you may install it to a MicroSD card using `mix burn` or upload it
to a device with `mix upload` or `mix firmware.gen.script`+`./upload.sh`.

Upload the firmware over the network (or burn a MicroSD card).

joel@Joels-Dell-XPS:~/hello_nerves$ ./upload.sh  172.31.52.93
Path: ./_build/rpi0_dev/nerves/images/hello_nerves.fw
Product: hello_nerves 0.1.0
UUID: fcc0965d-62d2-5fa9-f3db-77924ccb963a
Platform: rpi0

Uploading to 172.31.52.93...
fwup: Upgrading partition B
100% [====================================] 31.96 MB in / 34.21 MB out
Success!
Elapsed time: 16.492 s

Connect to the Raspberry Pi Zero W via ssh. Check the version numbers, firmware partition, network addresses, etc.

Interactive Elixir (1.17.2) - press Ctrl+C to exit (type h() ENTER for help)
β–ˆβ–ˆβ–ˆβ–ˆβ–„β–„    β–β–ˆβ–ˆβ–ˆ
β–ˆβ–Œ  β–€β–€β–ˆβ–ˆβ–„β–„  β–β–ˆ
β–ˆβ–Œ  β–„β–„  β–€β–€  β–β–ˆ   N  E  R  V  E  S
β–ˆβ–Œ  β–€β–€β–ˆβ–ˆβ–„β–„  β–β–ˆ
β–ˆβ–ˆβ–ˆβ–Œ    β–€β–€β–ˆβ–ˆβ–ˆβ–ˆ
hello_nerves 0.1.0 (fcc0965d-62d2-5fa9-f3db-77924ccb963a) arm rpi0
  Serial       : 00000000ae60f1ba
  Uptime       : 2 minutes and 26 seconds
  Clock        : 2024-07-11 07:02:36 UTC (unsynchronized)
  Temperature  : 36.3Β°C

  Firmware     : Valid (B)               Applications : 38 started
  Memory usage : 53 MB (17%)             Part usage   : 0 MB (0%)
  Hostname     : nerves-f1ba             Load average : 0.10 0.08 0.03

  usb0         : 172.31.52.93/30, fe80::ac91:dfff:fe14:edbd/64

Nerves CLI help: https://hexdocs.pm/nerves/iex-with-nerves.html

Toolshed imported. Run h(Toolshed) for more info.

Cross fingers for luck and list available cameras.

iex(1)> cmd("rpicam-jpeg --list-cameras")
Available cameras
-----------------
0 : imx219 [3280x2464 10-bit RGGB] (/base/soc/i2c0mux/i2c@1/imx219@10)
    Modes: 'SRGGB10_CSI2P' : 640x480 [206.65 fps - (1000, 752)/1280x960 crop]
                             1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
                             1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                             3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]
           'SRGGB8' : 640x480 [206.65 fps - (1000, 752)/1280x960 crop]
                      1640x1232 [83.70 fps - (0, 0)/3280x2464 crop]
                      1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                      3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]

0

Stream video from the camera to demonstrate that it works. The Raspberry Pi camera software documentation provides the commands. In iex, enter a shell command:

iex(10)> cmd("rpicam-vid -t 0 --inline --listen -o tcp://0.0.0.0:8888")
[0:32:52.216628000] [206]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[0:32:52.318085000] [207]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:32:52.337195000] [207]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[0:32:52.341804000] [207]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media1
[0:32:52.342112000] [207]  INFO RPI pipeline_base.cpp:1102 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'

On another computer open the VLC Media Player. Select Media/Open Network Stream from the menu. Enter the media stream URL tcp/h264://nerves.local:8888. VLC will display the delayed video stream after a few seconds. Meanwhile, on Nerves, rpicam-vid will log the connection and begin logging frames sent.

Preview window unavailable
Mode selection for 640:480:12:P
    SRGGB10_CSI2P,640x480/0 - Score: 1000
    SRGGB10_CSI2P,1640x1232/0 - Score: 1444.49
    SRGGB10_CSI2P,1920x1080/0 - Score: 1636.67
    SRGGB10_CSI2P,3280x2464/0 - Score: 2162.49
    SRGGB8,640x480/0 - Score: 2000
    SRGGB8,1640x1232/0 - Score: 2444.49
    SRGGB8,1920x1080/0 - Score: 2636.67
    SRGGB8,3280x2464/0 - Score: 3162.49
[0:41:48.022678000] [206]  INFO Camera camera.cpp:1183 configuring streams: (0) 640x480-YUV420 (1) 640x480-SBGGR10_CSI2P
[0:41:48.029428000] [207]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 640x480-SBGGR10_1X10 - Selected unicam format: 640x480-pBAA
#0 (0.00 fps) exp 33289.00 ag 8.00 dg 1.00
#1 (29.98 fps) exp 33289.00 ag 8.00 dg 1.00
#2 (29.97 fps) exp 33289.00 ag 8.00 dg 1.00
#3 (29.98 fps) exp 33289.00 ag 8.00 dg 1.00

When the VLC playback stops, rpicam-vid will terminate because the TCP connection is lost. I consider this to be a successful conclusion.

#231 (29.97 fps) exp 33289.00 ag 8.00 dg 1.00
#232 (29.97 fps) exp 33289.00 ag 8.00 dg 1.00
#233 (29.97 fps) exp 33289.00 ag 8.00 dg 1.00
terminate called after throwing an instance of 'std::runtime_error'
  what():  failed to send data on socket
134

For extra credit and in the interests of science, I experimented to learn which of these config.txt lines were necessary.

# Load the Raspberry Pi Camera 2 overlay
dtoverlay=imx219
camera_auto_detect=0

The table below shows whether libcamera detected the camera for various combinations of retaining, omitting, or altering the configuration lines. Libcamera detects the camera when auto-detection is enabled or the overlay is explicitly loaded.

dtoverlay=imx219 Omit dtoverlay=
camera_auto_detect=0 Detects camera No camera available
camera_auto_detect=1 Detects camera Detects camera
Omit camera_auto_detect= Detects camera No camera available

Most Liked

fhunleth

fhunleth

Co-author of Nerves

@jshprentz Thank you for this write up! We should definitely have drivers included for the official Raspberry Pi cameras by default. I put what I think is the minimum needed to support all official cameras at Include all RaspberryPi camera device tree overlays by fhunleth Β· Pull Request #342 Β· nerves-project/nerves_system_rpi0 Β· GitHub. Assuming this works, I’ll apply it to the rest of the 32-bit Raspberry Pi systems that were recently updated to libcamera.

Where Next?

Popular in Discussions Top

joeerl
I’m playing with Elixir - It’s fun. I think @rvirding does give Elixir courses these days. Re: files and database - when I given Erlang ...
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40042 209
New
marciol
Please, let me know if this kind of discussion already took place in another topic . Hi all, how do you consider if is better to build ...
New
fireproofsocks
I’ve been working on an Elixir project that has required a lot of scripting. I usually reach for Elixir because I like it more (and in th...
New
arcanemachine
https://nitter.net/josevalim/status/1744395345872683471 https://twitter.com/josevalim/status/1744395345872683471
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36654 110
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44532 311
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement