NervesHub - Delta updates

Nerves, fwup and NervesHub have had delta updates as a feature on the checklist for a long time. A delta update being essentially a patch you send to update the binary data on a device instead of sending all the desired data. It can be way more efficient in terms of bandwidth. We use xdelta3 for the binary diffing.

The delta support has been kind of gnarly though. In several important ways they were best-effort and whether they worked or not was dependent on what fwup features you used and how you used them. If you used disk encryption it was a no-go.

Thanks to work at SmartRent we now get fully functional delta updates that understand way more about what the heck you are doing with your firmware. And you don’t really need to think about it very much. Just turn on Delta updates for your deployment and if you have a compatible setup you should get the advantages. This is all on main in nerves_hub and you get it if you run the latest tag for the container or similar.

There are still updates we can do to Nerves systems to make delta updates apply to a larger part of the system. Currently they tend to apply to the rootfs.

A bunch of things needed to be lined up to make this happen:

• NervesHub needed to get smarter about generating the delta packages to understand whether an update was possible. Is the version of fwup high enough? Does this archive have support for deltas at all? Does it use disk encryption and does the delta update respect that?
• fwup needed to provide a combination of the disk encryption feature and the delta update feature. Awesome job by @fhunleth

fwup uses the libconfuse format for config files so as part of this I made a kind of rough library for dealing with those files, I call it confuse. A bunch of nimble_parsec and then some really helpful functionality to interpret the structure based on fwup usage and conventions. Most of the smarts around deltas for NervesHub live in confuse.

fwup itself does not know how to create a delta firmware archive, maybe it should but currently it does not. So NervesHub becomes the reference implementation so even if you want to use another delivery mechanism you can yoink the functionality out of it.

NervesHub will create deltas for the files that have the relevant delta options applied and other files will be provided in full.

For an extreme case where we just bumped the version number we actually saw a delta update down in the kilobytes for a firmware that was 90Mb. In reality it will vary significantly. One usually mild trade-off is that applying the firmware will likely take longer due to compression and the procedure of reading and patching the existing data to write to the new partition.

You can try this on NervesCloud already as we run very recent versions. Feel free to weigh in with questions in this thread or elsewhere.