Emerge & Solve - a GUI framework for Elixir

Your Libraries & OS Mentoring Libraries
, , , , ,
1

Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve.

They are GUI (Emerge) and State management (Solve) libraries that form a framework when put together even though they don’t depend on each other.

Currently only Linux is supported, to be precise Wayland and DRM (Nerves). MacOS and Windows support is planned sometime in the future. Tested only with AMD (Wayland and DRM), rpi5 and rk3566 nerves devices (DRM)

There is also a Solve LiveView adapter in the works, you will have to ask @ken-kost for the timeline on that.

If you want more details check out on hex and github. Rest of the post will be on hows and whys of development.

Getting artificial elephant out of the room

Before I get any further, this project was a successful experiment on how far I can push gen AI coding. 90%+ of the code in those repos is LLM generated and I could maybe credit it for about 5% of the engineering behind that code, there were sessions with 50+ prompts in plan mode before implementation. I have a pretty good understanding of how incapable AI is. That being sad I still have some coding standards and these projects don’t meet them but that is a future me cleanup problem. If this has made you lose all interest I fully respect that. However this post is 0% AI. I will not even use it for spelling & grammar. I have enough trauma of trying to use LLMs to write the docs and I still hate current state of docs. We can start a new thread if anyone wants more details on AI usage, I would like to keep this one focused on GUI.

Motivation

I use nerves both Professionally and all my hobby projects are tied to it, all of those require UI running on nerves devices and I need/want them to be both easy and fun to develop and high performance at the same time. I also have aspirations to develop a mix of OBS and real-time Davinci Resolve using this system on top of membrane, foundation for video pipelines is already in there.

I have spend half of my career building UIs. Using native Android/iOS, meteor.js, Elm, React, ReactNative, Flutter, Surface/LiveView, Keechma-next(Clojure/Script) in various combinations of state managements and different css frameworks/UI toolkits between all of it.

Most of these tools share a common trait. They are painful to use or turn painful to use as UI becomes complex, each set of tradeoffs falls apart in one way or the other. These 2 are most pleasant from both sides:

  • Elm-ui is only lib where I could basically one shot full page from the design without ever running it and it would turn out mostly correct. Emerge API is based on it. When used in Elm this lib becomes painful because elm pushes you into prop drilling.
  • Keechma-next had the most pleasant state management solution of them all. It achieves full decoupling of UI and state without prop drilling while having explicit and declarative data flow in a way that scales very well with complexity. Solve is an attempt at 1:1 elixir re-implementation.

It boils down to me wishing for a combination of those 2 every time I have to write an UI. Last time I was building a larger feature in flutter for nerves device just tipped me over and I have decided to just go for it.

Implementation

Emerge is mostly a Rust NIF backed by skia.

You declare a recursive struct in elixir via function based DSL. Struct is diffed with previous version in Elixir, diff is then serialized into binary and sent over to the Rustler NIF.

Rust side consists of the layout engine, renderer and event engine.
Different back-end implementations need to provide renderer with surface to render into and they need to provide output to the event engine.

Hope some of you find this project useful even as it is in this initial version. There is a lot of essential features still missing. More backends and features are coming in the future along with performance optimizations as there is some low-hanging fruit on that side.

If you try it out and are unable to implement a UI feature you want using Emerge feel free to reach out and propose a new feature.

20 Likes
Mob - Native BEAM / Elixir on Native Mobile
2

So far it looks really cool, do you have any examples which showcase most of the available features? What coding agents have implemented the library, what setup was used?

3

Checkout GitHub - emerge-elixir/emerge_demo: Example showcase app built with Emerge and Solve · GitHub. There is a showcase app in there that covers all of the components and attributes.

I tried lot of models and harnesses in the beginning, settled on Opencode + 5.3Codex, 5.4 later.

1 Like
4

So how much, if anything, is written by AI? 90%+, 5%, or 0%?

5

Sorry, words got lost there, edited. This post is 0% AI. Code is mostly written by AI but it ended up being more of an automatic text editor because I had to have it refactor same thing for 10 times and sometimes I had to do it from scratch in the end. For example there was no way to describe events system that I have imagined to AI, had to give up after hundreds of prompts. In the end it was still a lot faster then doing it by hand. Once there is concrete pattern to follow it can do the remaining 75% of implementation without a problem.

2 Likes
6

If you have rpi5 check out a demo project for nerves

You can iterate UI only changes by pasting UI module into IEx shell.
It will require an action that triggers viewport to rerender, for example hitting a + on counter.

That means you can deploy Emerge UI hotfixes using NervesHub on-connect code in production :upside_down_face:

1 Like
7

Solve 0.2.0 is out bringing solve oriented flow for handling general messages to controllers with handle_info/2..5

Emerge - 0.2.1 is out bringing initial support for macOS. Aim for this release is to give macOS users to develop nerves apps easily if you wish to develop proper macOS apps that you can distribute through app store you will have to come up with packaging and signing story. video_target is also not currently support on macOS

If you want to try it out on mac clone GitHub - emerge-elixir/emerge_demo: Example showcase app built with Emerge and Solve · GitHub and run with iex -S mix

1 Like
8

Emerge 0.3.1 is out compared to 0.2.1:

  • There is initial version of layout caching and renderer cache. Depending on app structure it can bring huge performance improvements especially on animated sections.
  • There is now Input.slider
  • Size.min/max are now proper size resolution combiners. For example now you can have an element with height(min(content(), fill()) attribute and it will be content sized until content becomes bigger than it’s fill portion. TodoApp in demo now uses it for entries so it grows with entries to screen size and then after that entries become scrollable since entries element has scrollbar_y on it.

Next roadmap:

  • Add headless backend, you give it pid and it sends rendered binary to that pid every time there is a change.
  • Make all backends automatically fallback to raster rendering in case there is no gpu.
  • Add option for output pixel format (HDR displays, 1 bit or 2 bit e ink displays)
  • Both of these will be preliminary work for running it on nerves badge/ nerves starter kit. Goal is to have next release run on it :slight_smile:
3 Likes
9

This looks pretty cool. I have also been working on a native GUI framework for elixir and gleam, and ended up with something much closer design-pattern wise to how a lot of elm loops work. So, it’s nice to see a different declarative approach with solve. For nerves, I ended up going with the idea of a remote renderer that could allow connecting over something like erlang SSH or regular SSH over stdio instead of running natively on the device. Gives some interesting possibilities for non-web server UI as well. Curious if you’re using NIFs or a separate process via ports and what your experience with that has been.

My project is called plushie (plushie and plushie_gleam on hex). It started off with the goal of supporting elixir and gleam for both local and remote desktop apps, but it’s taken on a bit of a life of its own and now supports other languages as well. The main thing I’ve been working on is around the renderer and build tooling to allow for custom rust to get mixed into projects similar to how rustler does things. There might be some things I’ve done as part of all that which could be useful to you in your own endeavors. I’m also not convinced that iced is the right underlying GUI library and have been considering something else like gpui-component, but there might be some opportunity here to work toward some common underlying renderer that could be shared between our projects.

3 Likes
10

Plushie looks interesting. I am not completely sure we can make common underlying renderer on any level higher than skia without making big compromises.

On NIF/port qestion currently I use both:

  • On linux it is Rustler NIF, UI declaration is sent to rust using internal binary format called EMRG and rustler sends messages of resolved events to pid setup when starting renderer. If you use it how documentation describes it it will all be setup by viewport.

  • On macOS I had to go for different route. macOS host is standalone rust program that is currently started as a port. macOS requires for main renderer loop to be on main thread. So BEAM now spawns macOS hos as a port and communicates with it through unix socket with yet another small internal binary format. It is archited that way so in the future you can have a macOS host start a beam and give it unix socket to have communication with main renderer loop. That will be needed for properly packaging macOS apps for the store. I think iOS will need similar approach, not sure what are Android limitations.

As I plan to rewrite internals before 1.0.0 (possibly multiple times, as it is a bit of a slopfest now) EMRG binary communication can be another layer to target for plushie but that would require plushie to compromise on stack, pin, floating and maybe some other widgets or very awkwardly implement them on top of emerge API. It would require you to implement something like current macOS host for every language. I do plan to add a few more widgets and some kind of canvas API in the future.

Currently EMRG is not very well documented or stable currently, there is overview here and elixir serializer here

I am aiming for performance with Emerge and that requires specific UI description model, as relayouting and caching boundaries. Goal is if you strap a touchscreen onto rpi running nerves ui should be as responsive as last flagship phone that had screen of same refresh rate. If you are running desktop app on 500hz gaming monitor there is no reason for ui not to run smoothly on 500fps (currently still fall a bit short on full page re-renders but I have a plan how to improve performance greatly in the future). And there is no reason for standard GUI app to consume more than 50 MB of CPU ram (looking at you electron apps). Lib size is around 6MB(excluding few system dependencies like gpu drivers and native graphics stack for each platform) for now and I will try to keep it from growing.

We can chat more or hop on a call if you want to discuss bit more.

2 Likes
11

Theoratically I should be able to render into a VNC server and make remote GUI applications accessible with a standard VNC client, right?

12

Yes, it will also require for rust NIF to have a way of piping keyboard/mouse inputs into renderers event system to be usable. I am aiming on making that API, it will require some time to make get it right.

Headless renderer opens up a lot of possibilities. Emerge already has video target on Linux that can receive DMA buffers using PRIME descriptors (basically a pointer into gpu memory with metadata of what that memory contains) and render it inside of an element. I plan for headless renderer to also be able to send Prime descriptors instead of RAW bytes.

That enables making pipelines of many Emerge renderers as well as easily making membrane elements with Emerge and keeping it all in GPU as zero copy pipeline with DMA buffers. Here is still very early work of my zero copy video decoder. I plan to expand it to work with all HW codecs certain hardware exposes both for decoding and encoding. It will take some time but I will release video demo once Windows support for emerge is made and when I figure out DMA equivalents on macOS and Windows so it can truly be cross platform.

That means you will be able to use Emerge in many different scenarios, for example:

  • to draw overlays on top of video streams on server, and also make it really fast (low latency) if your server has GPU.
  • Implement your own VNC like system on top of few membrane elements.
  • pipe output GUI output as a video into existing VNC server.
  • Bulid desktop OBS like clone fully in elixir.
  • This building blocks should even allow you to build something like DaVinci Resolve, where you split various tasks over multiple headless renderers that stitch it together so you can utilize all of the hardware available in parallel.

Since Solve as state management is not directly attached to any Emerge viewport/renderer that gives you freedom to have one Solve app to orchestrate many Emerge renderers and vice versa, one Emerge renderer to pull from many solve apps.

IMO it is better to run user facing UI directly on device that user interacts with than having user remote into some UI via VNC.

1 Like
13

hey @Damirados :waving_hand:
how does this compare to scenic?

14

My first attempt started by making on top of Scenic and Emerge is kinda spiritual successor to Scenic.

I made scenic driver on top of skia to keep it GPU accelerated (compared to current cairo efforts) and more portable with better text rendering.

From usage perspective scenic lacks layout engine and it’s state management is intertwined with UI.
I attempted to make layout engine on top of scenic but very quickly realized Rust is better suited language to write layout engine in than Elixir and I had ideas for event system, scrolling and animations where having Scenic in the middle just got in the way.

A lot of ideas are taken from Scenic (serialization of UI tree, multiple backends…).
Also, there is no way you can have high fps scrolling and/or animations if you drive them from elixir, crossing the NIF/port boundary is just too expensive.

I think Scenic is conceptually really cool idea but every time I tried to build UI in Scenic over past few years it felt hard, messy and took a lot of time to get any UI going. I guess that UX of using it is why it never got popular in the first place.

As Emerge is basically UI framework by myself to myself. It sits on a level of abstraction where it solves layouting scrolling, animating and event resolution but it still gives me control over everything else. With having 1 way to do the things like centering text into element or implementing a dropdown menu and without having any default styling or components (currently in text input selection highlighting is only default and not configurable part).

For example there is no checkbox or radio_button element since they are trivial to implement on top of Input.button, but there is Input.slider as that involves dealing with math that calculates widths of slider tracks and positioning of a thumb.

Basically if I sit down in the morning I like to have GUI app with dozen of screens by evening in more or less production ready state. And it hasn’t been like that since I was working in elm with elm-ui years ago.

I am still thinking if Emerge 2D canvas API should just be Scenic graphs since I already have a Rust lib that renders it with skia. If I go that way I will still make it in a way that discourages local state related to that UI.

2 Likes
15

Thanks for your reply!

We are currently using Scenic on top of Nerves to display a dynamic transparent overlay over an H.264 video (kiosk mode). It relies entirely on DispmanX, which is now deprecated and no longer works on recent kernels.

I’m wondering whether Emerge could help us achieve the same result, ideally with hardware-accelerated H.264/H.265 decoding.

16

I actually have a working demo of H265 decoding pipeline with membrane that displays dynamic transparent overlay, all zero-copy and GPU accelerated. I will try to clean it up and upload it as new emerge demo sooner than macOS video work is finished then, may take a week or two.

What nerves device are you currently using? Emerge currently requires GPU with atomic mode setting for DRM backend, that means rpi4 and rpi5 or mostly any arrch64 device that has working gpu drivers. in the future I will enable legacy mode setting to bring it to older devices and add precompiled arm 32 bit builds into release. There is a limit on how many things I can work on in parallel though so this may take some time.

2 Likes
17

We’re running on rpi3.

A solution working on rpi4/rpi5 would be fine, because eventually we’ll need to support these devices (how long will rpi3 still be produced?)

A solution working on rpi3 / rpi4 / rpi5 would be the ideal!

18

rpi3 does support atomic mode setting when using dtoverlay=vc4-kms-v3d. I do have one at home. I will need to test it out and see what changes are needed to the default nerves system. It may be a good time to try out GitHub - jimsynz/nbpr: Nerves Binary Package Repository · GitHub

2 Likes