How to use a non-supported model from huggingface in bumblebee or axon?

How do you load a non-supported model from huggingface in bumblebee? For example, Controlnet image-to-image model?

Or, since this is not a ‘transformer’ on huggingface, I think I’ll need to follow/copy the ‘stable_diffusion’ code and get more low level with Axon and Nx?