GES233
Building a ComfyUI-like system in Elixir
The origin of this idea goes beyond just building “Phoenix’s ComfyUI.” I want to combine BEAM’s parallel performance and fault tolerance with elegant syntax (even creating a friendly DSL), which are precisely the reasons I started learning Elixir.
Before posting, I explored existing FBP (Flow-Based Programming) related works. However, most packages either don’t fully align with my vision or are domain-specific (e.g., web-focused applications like Ash/Reactor).
While I welcome discussions about flow programming/job orchestration feasibility in Elixir, I want to ground this conversation in practical context to avoid purely theoretical debates.
My motivation stems from an ongoing project: building a singing synthesizer interface/app that integrates multiple models (initially aiming to implement DiffSinger’s Elixir wrapper with a simple WebUI).
For those familiar with song synthesizers like Vocaloid: generating audio from lyrics involves multiple steps and requires parameter adjustments at various abstraction levels (note duration, syllable timing, pitch curves, etc.). This complexity necessitates choosing different models/phoneme dictionaries, which inspired me to build a tool that lowers the usage barrier.
A ComfyUI-like system seems ideal for this scenario. A “track” could be defined as a workflow of interdependent tasks, where individual tasks might require operations from others.
I’ve created a repo under SynapticStrings/QyEditor: “Lightweight synthesizer interface.”. Currently, the project is primarily in Chinese due to my limited English proficiency and intentional delay in internationalization (docs/comments/etc.) until the core is stable.
The architecture splits into two applications:
:qy_corehandles parameter manipulation and chaining (similar to Plug’s philosophy).:qy_flowmanages parallelism, task scheduling, and process orchestration using libraries like GenStage/Flow.
Key questions:
- Is building a ComfyUI-like system in Elixir feasible?
- Am I on the right track with this architecture? What should be my next steps?
P.S. I know the optimal path would be to implement DiffSinger’s pipeline in Elixir first. However, calling its ONNX model via Ortex throws errors, and my limited Rust/ML debugging skills prevent me from resolving this.
Most Liked
VictorGaiva
It is very likely possible, but would require a lot of community work, and a few contributors.
Now even more possible with PythonX
asianfilm
It’s fine to use LLMs while coding but I don’t want to read posts here generated by AI. If you were overwhelmed, you could have taken a few days to think what you, as a presumed human, wanted to say. If the AI wants to post, there are forums for that.
Vidar
I’m not familiar with diffsinger, or the problem space, but I have used visual programming a fair bit over the years for various tasks. So I think I can say something about that part at least. Indeed I have earlier mused that many Elixir programs, with their modules and connections visualized with a Mermaid graph, could look very familiar to visual programming.
I would not choose visual programming for the UI alone. It can be very elegant with small programs, but for big programs it can easily turn into real life visual spaghetti code. At which point making components, color coding and or framing entire areas of code with what they actually do is a good idea. Then you have bigger components which again hides detailed complexity and allow easy overview.
My experience is that visual programming have some strong points:
- Isolation, reuse and making components. This is just trivial, and the reuse is real.
- Refactoring. Again, often just trivial cut and paste and move around.
- Parallelism. You can make independent parallel paths of visual code and you will get programs that runs that code in parallel. (Depending on actual hardware, and not all visual programming languages do this equally well). Back in the day I did FPGA programming using Labview, and it was like painting code onto actual hardware.
- UI. You see your program flow very easily. Depending on the language programming concepts like loops can be less elegant or just not possible at all.
- Debugging. Issues are often highlighted at the exact spot or module. Data comes in, but data doesn’t come out. Making a type breaking connection is usually impossible and thus found before even trying to compile. (Again, depending on language).
Like Elixir visual programming is also about transforming data, through visible steps and with connections between independent modules. Like functional programming the variables don’t change so if you want to reuse the value of a variable from earlier in the graph just make a connection to it at that stage.
The bypass caching & incremental generation you mention is a necessity during development of big graphs, or any minor change will end up triggering entire graph recalculations.
I don’t see why the internal architecture of components would make a visual UI for composing them any harder? The components should be independent, and with explicit inputs and outputs only, so what goes on inside is just their business. Indeed many visual languages make it easy to use different languages inside of components to better suit the components task. For composition the input and outputs, with types and data shapes, should be enough.
Popular in Discussions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









