The base layer and the application layer are separated. Setup/ drivers for the GPU, NPU, VPU and VintageNet are in the base layer. There might be a few dead remnants of from earlier Collabora and NPU poking adventures. Adjust to taste.
I’ve packed up a zip file with the base layer to make a small standalone package. Running a isolated test build now to make sure it is a clean standalone.
Not really as this is just the test setup with somewhat representative test stand-ins for now. I’ve found an old git account I can use.
The base layer only build failed at first, so the clean separation wasn’t clean enough it seems. And while I was at it I thought I would replace some Radxa specific boot files with mainline buildroot/ linux sources so there is less special Radxa sauce. That didn’t go too smoothly at first, and then Radxa had gone and changed their kernel under my feet so the link was dead.
Anyway, I’ve tried to cut back to what it takes to build the basic from basic source and avoid blobs as much necessary. Those necessary are linked to. I did a test build and it boots up, so classical it Works on my machine. I haven’t tested the GPU NPU and VPU on this one, but the same setup do work on the larger test build. It can be found at GitHub - BadBeta/base_radxa_rock_5t_nerves_GPU_NPU_VPU: Base Elixir Nerves build for Radxa Rock 5T with RK3588 GPU NPU and VPU setup · GitHub
There were some leftover after the NPU exploration adventure and the changing of boot files messed up boot. Both should be fixed now in a fresh git push. Apart from the flashing to the emmc, which require Rockchip tools, it should be the normal mix nerves commands for building and flashing. (Although I’ve only tested with the emmc so here is too hoping!)
Awesome. Thanks for sharing. Gotta find some time to see what it would take to get on the Orange Pi 
Added a confirmed working HDMI input today, and some info on how to boot from Micro SD instead of Emmc.
Updated with ISPs, a 40 IO pin setup with [I2C, SPI, PWM, ADC, serial, IO], PDM microphone contact, MIPI cameras, MIPI display, various camera models compiled into the kernel, more video encoding and decoding codecs.
I will not be able to check all until I’m back at the electronics and have some time, but it does boot, and all I can check works.
With the exception of the cryptographic, compression and voice detection hardware I think most is covered.
I’ve started on an Elixir library to provide a layer on top of the hardware so I have access to all from Elixir land. Using circuits is a given for some IO functions, but for the things like the ISP, NPU and VPU it is less obvious how to make them Elixirish and user friendly. (Gstreamer and ffmpeg haunts me!).
The ISPs have a recipe you decide up front for each. From there it will apply that recipe to every Bayer Raw image or frame of video stream you apply it. Thus to use it later just a call to ISP(which one) seems fine to me.
For the NPU it is more complicated as there are so many different uses and models. But for inference calls on frames in a video stream I think it makes sense with a pass through piping where the just NPU grabs a copy as often as it can. For the RGA, which does overlay among other things, there need to be just a reference to the code to be used. The VPU output is basically just output size and codec. So that can be inline.
Thus for taking a video source like a CSI camera, doing some ISP noise reduction and contrast, NPU Yolo inference, adding rectangles overlay over detected objects, and using the VPU to transcode to 1280x720 h265 output would look like:
Video.from({:v4l2, “dev/video0”})
|> ISP.apply(1)
|> NPU.infer(Yolo8, objectpos_buffer)
|> RGA.overlay(rectangles, objectpos_buffer)
|> VPU.transcode(“/data/out.mp4”, :h265, {1280,720})
I think this looks ok and readable while still saying what is going on. It also makes it possible to keep track of which accelerated hardware is being used. They have capacity limits so that helps to not accidentally use the same hardware twice. (All the above is accelerated hardware only. There should hardly be any CPU activity at all).
The ISP call could maybe be improved by aliasing the ISP number with a sensible name describing that it does like ISP.apply(noisereduction_contrast), but then the coherence of that alias would have to manually tracked to match the recipe used. It might end up misleading instead. And harder to keep track of used hardware.
The various kinds of NPU models with all their different inputs and outputs is a bit of a headache to do cleanly though. Or maybe I’m overthinking it?
Wow, you are tooling this thing up!
Would that example pipeline just be passing memory pointers or controlling a NIF? Or are you actually commanding the devices from Elixir?
For pipelines like this I think Membrane is an answer but you can always build a Membrane pipeline on top of a good base library but you can’t always build the tightest possible thing if you only have Membrane elements.
I think the Nx ecosystem has a fair number of references for how they do pointers and on-device stuff with NIFs and such to avoid copying back and forth. It has been a topic in ADBC + Explorer, in Pythonx + Nx and probably in EXLA/EMLX.
But essentially the Elixir code ends up passing around some kind of opaque reference that represents a device and memory location that is owned by a NIF typically.
The telling things when to pull a frame and where to throw it to can be pretty good in Elixir from what I’ve seen and having access to that from Elixir is ideal. But being able to tell it to just go at full clip is probably a good thing to have as well.
My plan is sadly more boring and basic. I just use Zigler to wrap all commands of the existing C libraries from Rockchip to use these hardware accelerators.
These libraries are already meant to work together in a hardware pipeline from Rockchip. Each of the pipe elements is a hardware accelerator and the handoffs between them are done as zero copy using the processors purpose made DMA buffer. The RGA draws the Yolo rectangles directly onto the DMA buffer.
The only odd duckling out is the initial use of the ISP before the NPU. That seems less pipe integrated. The ISP takes a Bayer RGB input, and outputs a normal picture frame, while the NPU wants an interleaved data structure. So there are two options to solve that: 1. Modify the input of the AI models to take the ISP output as direct input. No conversion needed in the pipeline then, but a big hassle and likely causing the odd error. Or 2, which I have chosen, to use the RGA accelerator to do the conversion. The RGA should have no issue keeping up with the max 60fps flow rate (a limit of the 2x30 fps ISPs), but it does add a bit of latency.
When the NPU wants a new frame that has to be resized to model input before storing as a copy in the NPU ring buffer. The RGA does that too. The NPU setup is 4 frame positions for 3 NPU cores, so one is always being loaded so a new frame is ready when a core needs it. (That architecture takes 3 times more memory , so maybe making an option to use 3 cores for 1 frame. That apparently has less overall throughput though).
I’ve used Membrane some before but I wouldn’t say I’m really familiar with it. It is more like keyboard pecking, observe and hope for the best. Rinse and repeat until tests are happy. Over time some more systematic wisdom might accumulate. I plan to pipe the video output into Membrane to stream the output but still work to be done implementing the above.
References is the way here for sure, and the libraries and their DMA buffers are doing zero copy all along the hardware pipeline with exception for the sideline with NPU grab with resized copy. To get the main pipeline output from the DMA buffer at the end I currently also do a copy. I like to think there is a better solution to be found.
The NPU on the RK3588 is not supported by the XLA, so by extension I figured likely not by EXLA or NX either? That was my initial thinking anyway. And then I found that this 3588 has hardware accelerators sticking out everywhere so I thought I’d try to make it possible to compose them, line them up and use them from Elixir.