What is the recommended method for adding a custom linux kernel driver to a nerves device?

I have a custom hardware device that I would like to add to my nerves device. I already have created a custom nerves device (based on rpi3a) per the documentation, but I need to go further and add a custom linux kernel that does not exist in the kernel source. I forked nerves_system_br and started adding my own packages, but this feels like a hack, and will be hard to keep this solution linked with the upstream changes. Is there a better way?