Vidar
I’m putting this in its own thread as that makes more sense.
The little prehistory here is that I have a RPI5 system with Hailo 8 working, but the non AI/ video part is not quite where we want it. The Rockchip 3588 processor, with SBCs often compared to the RPI5, seems like a more capable beast. It has twice the number of processor cores, but more importantly it has a built in NPU and very capable VPU. It also have other specialized hardware which might be of interest such as GPU, jpeg encoding and decoding, 48MPx ISP with functions like [auto focus, HDR, RAW conversion, lens corrections], 8K hdmi output, 4K hdmi input, more audio functions than I’ll ever need, various IO, and then some. So I got a Radxa 5T version, which also comes in an industrial version, with the full intent of running Nerves on it in anger.
Getting this up and booting with Nerves was quite straight forward. There are two different kernel options:
-
The manufacturers with most hardware working fine but with a custom older kernel (6.1) and many proprietary blobs for drivers. (The NPU driver itself is actually open source). Issues I ran into with this one was getting WiFi working (the Radxa 5T SBC I got has a newer chip than 6.1), and getting a GPU accelerated browser for a kiosk mode using the Mali610 GPU. I backported the WiFi so that was fine, but getting a GPU accelerated browser was not so easy.
-
I then tested the other alternative, an open source 6.18 kernel with patches for the 3588 made by Collabora with Mesa3D graphics, Rocket driver and Teflon TFlite delegate. I believe many of these or maybe all (?) are now in mainline Linux. Anyway, getting Nerves booted was troublefree, and I also got a GPU accelerated Chromium running in kiosk mode fairly easily. Headphone sound is a bottomless mystery to me though so that is not working yet.
The problem with this is the NPU and VPU rely on the open source drivers and kernel to work. Teflon is young and has basically just implemented the operations needed to run their test Mobilenet (if I remember correctly). I’ve written more in another thread about trying to get these open source alternatives to work with Yolo 8, and in the end I basically explored the NPU registers and poked them directly. (The LUT was surprising as both Rocket and the official documentation had the number of LUT values wrong. Further, rather than being an actual LUT table it is more like a look up interpolated graph). In the end I found that the Rocket NPU initialization does not seem to support all that is needed for all operations on the NPU, and poking registers directly was hardly a good solution. So not really a way forward.
At that point I thought I could take the original Rockchip open source NPU driver and patch that into something that would work with the open source 6.18 kernel. But it seems I can’t do that without also having to change the 6.18 kernel (which is adopted to Rocket I presume). That actually makes sense as Rockchip themselves also had to fork the their kernel from the mainline to get their combination working. But if I changed the 6.18 kernel then other parts depending on it might accidentally break now or in the future. And all this was just for the NPU… I also wanted the VPU working, and the open source alternative is not full operation there either. A better plan was needed.
So, back to square one. Instead of trying to wrestle the Rockchip NPU and VPU drivers to fit with a 6.18 open source kernel the better plan was to keep the older Rockchip 6.1 kernel with the hardware working. And then somehow massage the Mali610 GPU driver blob into providing browsers with GPU acceleration.
Internetting I discovered that the key issue had been narrowed down to some missing communication between the browser and the renderer. And someone had even made a fix! But after applying the fix the speed was just 20 fps in the browser. The hook fix did make the GPU work, but it solved it by copying frames from the GPU to the CPU and then to memory for the renderer. Hence the low speed.
The better solution would be a zero frame copying by using references instead. So far that seems to work. The browser is now GPU accelerated at around 60 fps. Video playback in the browser should be accelerated by the specialized VPU though, so that is next on the list.
So far so promising.
Where Next?
Trending in Discussions
Other Trending Topics
Latest Nerves Threads
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 28 Posts
gBillal
Nice to see someone actually experiment with RK3588. I did create a nerves system for rock 5B+ too and since I’m mainly interested in the VPU my only option for the kernel is rockchip 6.1 since as far as I understood there’s no drivers in the mainline kernel.
I also got an issue with the wifi and I included an out of tree driver in the buildroot config, the issue is I still need to load the kernel module manually otherwise the wifi is not detected. Did you manage to actually make it work without manual intervention ?
I hope to get some time to check the GPU & NPU.
Damirados
You can track mainline kernel progress here. VPU is being worked on 7.0-rc1 has h264 and h265 decoding enabled VP8 and AV1 were done previously but AV1 need some IOMMU work still. VP9 is in progress. Below the table is link to collabora WIP kernel with all the pending changes merged already so for the time being it may be the best one to use.
NPU also has some cleanup work pending
Is any of your nerves systems publicly available. I got couple of rock 5B+ boards since my project needs VPU both encoding and decoding as well as NPU. I am planning to make nerves system that is updated as new things are merged but it would be nice to just take one as a starting point.
Vidar
It boots just fine with WiFi enabled. I have the login hardcoded as laziness right now, but I also made a menu in the kiosk for searching for networks and adding, changing or saving the WiFi.
I spent an entire week with the Collabora, Rocket and Teflon as I did want to use the most up to date open source alternative. But I also want both an accelerated browser and full NPU working that just did not seem to be there yet. Hence the choice to stick with working hardware in the official Rockchip kernel and rather massage missing bits to work with that.
Having a go at the VPU functionality now by installing Gstreamer for testing. Hopefully also get it working through v4l2. We will see.
gBillal
The nerves system is available here: GitHub - gBillal/nerves_system_rock_5b_plus: Nerves base image for Radxa Rock 5B+ · GitHub
Vidar
Thanks for the link.
After a week I got the Yolo 8 working on the NPU by dropping Teflon and sidestepping Rocket for everything except initialization. I had working LUT activations, working multi-surfaces, working fused operations and interleaved outputs. But as that clean up link says there are hardcoded bits that was even in the initialization of the NPU which forced many operations to the CPU. I got down to 3-4 fps with many operations forced to the CPU. The official NPU driver should get about 30-40 fps for the same.
Hopefully Rocket and Teflon will get there, but I need this working soon, and there still seems to be some way to go.
I plan to share a few versions once I’ve tested them more. A clean boot one, a kiosk one, and one with the all the hardware I got working by then in it.
Vidar
Thanks for that. There seems to be some parts there I haven’t gotten to yet so that should be helpful.
Did you get the headphone sound working? Mine currently just make hizzing sounds at me..
Vidar
I just discovered that the headphone sound suddenly works. As I didn’t actually test for that lately I can only assume that the unexpected fix came along with the change of kernel. I’ll take it either way.
VPU also working now. 200+ frames per second transcoding some 4K h264 video to 1280x720 h265 format including tone mapping.
The 3588 have an ISP which, if I read the specs right, can do at least 2 x 30 pictures per second and apply various frame/ picture treatments and conversions. It would be interesting to link that up for frame processing right after a video input and before AI processing.
NPU is also working. Yolo v8s 640x640 is currently at about 16 fps, but not really optimized and with Whisper (sometimes) running too. Still some cleaning, polishing and fanciful language needed. But progress for sure.
lawik
Is this in the form of just buildroot or do you have a prototype Nerves system?
I’d love to see pre-release or current state systems. They don’t have to be tidy or have compiled releases. Whatever progress you have would be super to be able to look at
I am very tempted to pull my Orange Pi 5 Plus off the shelf. It has an HDMI Input even..
Vidar
I’ve got a Radxa 5T (also with HDMI input) running Phoenix Liveview with a slightly patched Webkit/ Cog for kiosk based GUI. I tried Chromium earlier with the 6.18 Collabora kernel and that seemed more responsive, but I thought Cog might be more suited for this. I might have been wrong.
For testing Yolo and Whisper I figured a Youtube page would provide plenty of varied testing grounds. Which is true but with DRM protection making it a bit more complicated. Current status are buttons for activating Yolo and Whisper on the videos. Yolo got the typical rectangle outlines and text with category and confidence on the videos. I had to go to Yolo 8 with FP16 as the precision of the converted 8 bit was off for some reason. Yolo dropped to about 10 fps. Whisper is confirmed to work (sometimes) in the background, but with a quite delayed start. It seems I’ve also broken the earlier scrolling transcription overlay somehow, so that is the focus right now. I want an almost live transcription so still some way to go on that.
Looking at that juicy ISP next I think, but that also seems less documented. But being able to process frames for contrast, color and sharpness before the AI stage seems quite attractive indeed. It might also be interesting to abuse the lens correction into making only the center of the frames sharp with gradual drop off towards the edges.
Later I will test and or wrestle the I2C and PWM outputs with various sensors and stepper motor drivers, and maybe the MIPI camera and display. I’m not in the right location for that though right now.
This test setup just passed 1GB in size, but if you want to have a look at this stage you are more than welcome. I’ve probably trampled all over normal Nerves conventions with sheer ignorant joy! Not sure if the setup it is directly transferable to the Orange version but it should be close.
The shelf is no place for a perfectly good Orange Pi 5 Plus.
lawik
But you have the separation of a nerves system and a nerves application/firmware project? I’d love to try all of it but most interested in having the foundation of a nerves system