Project: The Grand Kiosk, the Seeed Studio ReTerminal DM with Nerves

Nerves Forum Nerves Chat
1

I’ve mentioned it previously in this thread that I’m poking and playing with the Seeed Studio ReTerminal DM which is a $300-400 piece of industrial kiosk kit. I like the look and feel of it and it makes for a very promising prototype platform for anyone wanting to do visual or interactive stuff with Nerves.

Tom Winkler at Formrausch made a Nerves system for it. And I’m hacking away at a demo-friendly featureful sample project.

Figured I’d show some of my progress here on the forum.

I’ve posted YouTube Shorts about it as well (1, 2, 3).

It comes with a solid set of peripherals:

  • Touchscreen
  • WiFi antenna
  • Light sensor
  • Microphone array, two mics
  • NervesKey chip (ATECC608)
  • Convenient boot switch for flashing the CM4 inside
  • Slot for M.2 drive

Plus plenty of ports and some GPIO stuff. It can also be enclosed to make it reasonably IP-classed for water and dust supposedly :slight_smile:

A variant exists that gives you PoE + a camera. Wish I’d grabbed that, it is not upgradable to it.

I will try to put notes about what I do with this thing as I go :slight_smile: Feel free to ask questions if you follow along.

6 Likes
2

Tom got most of the peripherals working via Linux drivers. So I can query a bunch of driver “files” to get light levels for example. But the mic was not working.

arecord -l did not give the expected device information.

So I’ve had a spelunk all across the Seeed studio overlays, added new firmware versions for RPi. Poked alsa settings. Then found a missing piece of linux config. Added that.
Suffice to say I have the microphone working on the ReTerminal DM.

And also, I’m fairly certain everything except that piece of linux config can be removed. Verifying that now…

3

Yup. One line.

4

I spent a bunch of time hacking away on a test script for the VAD ONNX thing that should give us the ability to know when the microphone(s) detect speech and only then do cool things with it. Should prevent picking up noise and doing unnecessary continuous compute. The current versions of the model are no good for those instructions. I still gotta get this working.

The next step would be running Whisper on the chunks that are speech. And that I already know I can do.

Also the ReTerminal DM has an M.2 slot and I believe Seeed Studio supports it and the ReComputer with the AI kit. They have benchmarks. So if/when we get AI Kit support into Nerves this workload could hopefully be accelerated.

1 Like
5

I did get the necesseray VAD bits working outside of Nerves and documented it:

I am a bit stuck on a cross-compilation issue for Ortex that I’m not Rust-qualified to figure out. Seems like something generates a crap flag further down. I don’t think I even need the HTTP client that the failing TLS dependency uses.

So close :smiley:

2 Likes
6

Thanks to @abelino fixing my cross-compilation woes I can now run ONNX-based inference with Ortex on Nerves.

So I have done some Voice Activity Detection with Silero VAD on this branch: GitHub - lawik/nerves_cloud_kiosk at voice-activity

I made a Membrane pipeline based on the previous blog post and my old Lively LiveViews with Membrane talk.

It didn’t come out well in a phone video but I have a waveform going with LiveView and tint the screen edges whenever it hears voice activity. Fun stuff :slight_smile:

The microphone in this thing is very noisy. I wonder if I can do anything to improve that.

1 Like
7

Böööp!

2 Likes