I’ve been messing around with image generation models recently and thought this could be wrapped in a library or people could just use the techniques suggested in the blog post I wrote:
We can have commercial grade modern image generation without too much difficulty by running the underlying Python libraries. This is certainly lower hanging fruit than trying to port everything over to Nx and we can use Nx as the bridge to share memory via Pythonx.
I think after having a day to stew on it I’m souring on the idea of creating the library because it’s unclear anyone actually wants this. The people it will benefit can just as easily use a commercial API. Also, these models will go off the boil soon enough and then we’re stuck maintaining an obsolete library.
So for the time being I’m more than happy for people to read the above, glean the lessons I’ve gleaned, embark on some experiments if the ideas in the blog catch your fancy and copy the techniques I’ve outlined.
One more thing to mention is that Pythonx just added sending messages back to Elixir from Python so now with very little modification you can update your Elixir on the status of whatever is happening in Python. For now you’ll have to use the main branch in GitHub - livebook-dev/pythonx: Python interpreter embedded in Elixir and in the future it will be available in pythonx 0.4.8+.
In my example I had to resort to something a little more convoluted to get a similar result which was to wrap the Python code in a genserver and pass its state back and forth which does work and has other advantages but it is more involved.
I was looking forward to running Flux on my laptop So, yeah, I want such a library! I use AI to generate images for a side project (custom avatars for users), and a library, even incomplete, would be super useful. Thanks for the great work!
I discussed Bumblebee in the blog article in the original post. I’m suggesting Bumblebee is the wrong layer to abstract and perhaps PyTorch is more adaptable to the flood of changes.
If you just want to run Flux on your laptop you can clone that repo and run it. Depending what your laptop is it may or may not work.
This seems pretty cool to me. Have you played around with ComfyUI at all?
I was thinking it’d be neat to have an Elixir library to power complicated image generation pipelines like that, but it’s a huge amount of work.
Maybe a lot less if using Python for the core bits with this approach, though.
I have looked at ComfyUI a bit. This is the nature of this type of problem is that everyone wants something slightly different out of it so if I build what I want that doesn’t necessarily accommodate what others want and so I may as well have just made a demo project.
My son and I watch Corridor Crew all the time so we see them using ComfyUI for some cool projects. It’s kind of at the threshold of what my machine can handle for what I want to do so he would really want to do the project for me to go ahead with that.
So, yeah, I want such a library! I use AI to generate images for a side project (custom avatars for users), and a library, even incomplete, would be super useful. Thanks for the great work!
