Yes, it will also require for rust NIF to have a way of piping keyboard/mouse inputs into renderers event system to be usable. I am aiming on making that API, it will require some time to make get it right.
Headless renderer opens up a lot of possibilities. Emerge already has video target on Linux that can receive DMA buffers using PRIME descriptors (basically a pointer into gpu memory with metadata of what that memory contains) and render it inside of an element. I plan for headless renderer to also be able to send Prime descriptors instead of RAW bytes.
That enables making pipelines of many Emerge renderers as well as easily making membrane elements with Emerge and keeping it all in GPU as zero copy pipeline with DMA buffers. Here is still very early work of my zero copy video decoder. I plan to expand it to work with all HW codecs certain hardware exposes both for decoding and encoding. It will take some time but I will release video demo once Windows support for emerge is made and when I figure out DMA equivalents on macOS and Windows so it can truly be cross platform.
That means you will be able to use Emerge in many different scenarios, for example:
- to draw overlays on top of video streams on server, and also make it really fast (low latency) if your server has GPU.
- Implement your own VNC like system on top of few membrane elements.
- pipe output GUI output as a video into existing VNC server.
- Bulid desktop OBS like clone fully in elixir.
- This building blocks should even allow you to build something like DaVinci Resolve, where you split various tasks over multiple headless renderers that stitch it together so you can utilize all of the hardware available in parallel.
Since Solve as state management is not directly attached to any Emerge viewport/renderer that gives you freedom to have one Solve app to orchestrate many Emerge renderers and vice versa, one Emerge renderer to pull from many solve apps.
IMO it is better to run user facing UI directly on device that user interacts with than having user remote into some UI via VNC.
