GES233

GES233

Building a ComfyUI-like system in Elixir

The origin of this idea goes beyond just building “Phoenix’s ComfyUI.” I want to combine BEAM’s parallel performance and fault tolerance with elegant syntax (even creating a friendly DSL), which are precisely the reasons I started learning Elixir.

Before posting, I explored existing FBP (Flow-Based Programming) related works. However, most packages either don’t fully align with my vision or are domain-specific (e.g., web-focused applications like Ash/Reactor).

While I welcome discussions about flow programming/job orchestration feasibility in Elixir, I want to ground this conversation in practical context to avoid purely theoretical debates.

My motivation stems from an ongoing project: building a singing synthesizer interface/app that integrates multiple models (initially aiming to implement DiffSinger’s Elixir wrapper with a simple WebUI).

For those familiar with song synthesizers like Vocaloid: generating audio from lyrics involves multiple steps and requires parameter adjustments at various abstraction levels (note duration, syllable timing, pitch curves, etc.). This complexity necessitates choosing different models/phoneme dictionaries, which inspired me to build a tool that lowers the usage barrier.

A ComfyUI-like system seems ideal for this scenario. A “track” could be defined as a workflow of interdependent tasks, where individual tasks might require operations from others.

I’ve created a repo under SynapticStrings/QyEditor: “Lightweight synthesizer interface.”. Currently, the project is primarily in Chinese due to my limited English proficiency and intentional delay in internationalization (docs/comments/etc.) until the core is stable.

The architecture splits into two applications:

  1. :qy_core handles parameter manipulation and chaining (similar to Plug’s philosophy).
  2. :qy_flow manages parallelism, task scheduling, and process orchestration using libraries like GenStage/Flow.

Key questions:

  1. Is building a ComfyUI-like system in Elixir feasible?
  2. Am I on the right track with this architecture? What should be my next steps?

P.S. I know the optimal path would be to implement DiffSinger’s pipeline in Elixir first. However, calling its ONNX model via Ortex throws errors, and my limited Rust/ML debugging skills prevent me from resolving this.

Most Liked

VictorGaiva

VictorGaiva

It is very likely possible, but would require a lot of community work, and a few contributors.

Now even more possible with PythonX

asianfilm

asianfilm

It’s fine to use LLMs while coding but I don’t want to read posts here generated by AI. If you were overwhelmed, you could have taken a few days to think what you, as a presumed human, wanted to say. If the AI wants to post, there are forums for that.

Vidar

Vidar

I’m not familiar with diffsinger, or the problem space, but I have used visual programming a fair bit over the years for various tasks. So I think I can say something about that part at least. Indeed I have earlier mused that many Elixir programs, with their modules and connections visualized with a Mermaid graph, could look very familiar to visual programming.

I would not choose visual programming for the UI alone. It can be very elegant with small programs, but for big programs it can easily turn into real life visual spaghetti code. At which point making components, color coding and or framing entire areas of code with what they actually do is a good idea. Then you have bigger components which again hides detailed complexity and allow easy overview.

My experience is that visual programming have some strong points:

  1. Isolation, reuse and making components. This is just trivial, and the reuse is real.
  2. Refactoring. Again, often just trivial cut and paste and move around.
  3. Parallelism. You can make independent parallel paths of visual code and you will get programs that runs that code in parallel. (Depending on actual hardware, and not all visual programming languages do this equally well). Back in the day I did FPGA programming using Labview, and it was like painting code onto actual hardware.
  4. UI. You see your program flow very easily. Depending on the language programming concepts like loops can be less elegant or just not possible at all.
  5. Debugging. Issues are often highlighted at the exact spot or module. Data comes in, but data doesn’t come out. Making a type breaking connection is usually impossible and thus found before even trying to compile. (Again, depending on language).

Like Elixir visual programming is also about transforming data, through visible steps and with connections between independent modules. Like functional programming the variables don’t change so if you want to reuse the value of a variable from earlier in the graph just make a connection to it at that stage.

The bypass caching & incremental generation you mention is a necessity during development of big graphs, or any minor change will end up triggering entire graph recalculations.

I don’t see why the internal architecture of components would make a visual UI for composing them any harder? The components should be independent, and with explicit inputs and outputs only, so what goes on inside is just their business. Indeed many visual languages make it easy to use different languages inside of components to better suit the components task. For composition the input and outputs, with types and data shapes, should be enough.

Where Next?

Popular in Discussions Top

Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
MarioFlach
Hello, I want to share a project I’ve been working on for a while: https://github.com/almightycouch/gitgud Background Some time ago I ...
New
Fl4m3Ph03n1x
Background This question comes mainly from my ignorance. Today is Black Friday, one of my favorite days of the year to buy books. One boo...
New
mmport80
I have put far too much effort into Dialyzer over the last year or so - and basically - I doubt it’s worth the effort. It’s not as easy ...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
tmbb
This is a post to discuss the new Phoenix LiveView functionality. From Chris’s talk, it appears that they generate all HTML on the serve...
342 18146 126
New
fireproofsocks
I’ve been working on an Elixir project that has required a lot of scripting. I usually reach for Elixir because I like it more (and in th...
New
und0ck3d
Hello everyone! A few days ago I’ve created a topic here about how people were creating CMSs with Elixir and Phoenix. I’ve been studying...
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
New
griffinbyatt
Sobelow Sobelow is a security-focused static analysis tool for the Phoenix framework. For security researchers, it is a useful tool for g...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29377 241
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New

We're in Beta

About us Mission Statement