udoschneider
I’m struggling to understand a problem I’m facing, and I’m not sure if there’s an existing solution.
In the project I’m working on, we need to create embeddings using Bumblebee in conjunction with pgvector. Currently, all nodes in the cluster are running the same release. However, with the new requirements for embeddings generation, I want some nodes to have more powerful GPUs. These nodes will detect the presence of a GPU using an environment variable and will start the Bumblebee Serving.
The core idea is that all nodes can request embeddings generation, but the actual worker processes will only run on the GPU nodes.
I’m looking for a solution that combines a distributed registry with something like Poolboy, which would allow for round-robin or other load balancing on the registered processes. I’ve looked into HordeRegistry, but it seems to be designed for single processes only.
I have some ideas for building a custom solution from scratch, but I have a feeling that my requirements—such as service discovery and load distribution—might be met with an off-the-shelf solution.
Do you have any pointers?
Thanks,
Udo
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Elixir, Erlang & messaging conferences from Code Sync. Next up: ElixirConf US (10–11 Sep) & Code BEAM Europe (21–22 Oct), Haarlem + online.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
hubertlepicki
I’d default to Oban, and run some queues only on selected nodes.
udoschneider
In my opinion, the main limitation is that we need embedding generation to be available as a function call. However, you bring up a valid point—having dedicated queues specifically for the GPU nodes and wrapping the job generation in function calls that wait for job completion could be a solution.
hubertlepicki
Check out Relay then
udoschneider
Didn’t know about the Relay Plugin. Maybe going ObanPro is the way then …
hubertlepicki
it’s pretty good. Make sure the latency is acceptable for your use case. The performance should be there but sometimes there’s a little bit of latency before the job starts executing.
Horde could also work. I used it for similar use case with not Horde.Registry but Horde.DynamicSupervisor — Horde v0.10.0 for starting supervised processes and Horde.Registry — Horde v0.10.0 for their name registration and lookup.
In my experience Oban approach will be easier to get working right, without weird bugs.
udoschneider
Regarding
Horde.DynamicSupervisor, I initially explored it but decided against using it because all the distribution strategies I encountered aimed to distribute tasks among all nodes. However, I just realized that by implementing my own strategy using theHorde.DistributionStrategybehavior to distribute tasks specifically among GPU nodes, I might achieve what I need. This sounds intriguing—thanks for the suggestion!hubertlepicki
Not to intentionally derail your efforts but just come up with another idea. I have not used Flame but it kinda seems to fit your use case. I don’t know if there are backends that would suit you, however, and I never used it. Have a look at least:
https://github.com/phoenixframework/flame
udoschneider
Feel free to derail as much as you want; my line of thinking was already stuck.
Yes, I did consider FLAME, but I ultimately ruled it out because of the usage pattern. We need to generate embeddings consistently and predictably. My impression of FLAME is that it’s mainly designed for “spot” functions, where the startup time is minimal compared to the overall runtime.
hubertlepicki
sir, please do not encourage me. I ended up discussing medieval literature and Don Quixote in particular the last time. On this forum.
jstimps
I think you could get it done with a Ra state machine. The state machine could straightforwardly be the distributed registry. You can add in the load balancer piece via custom logic in the query path.
https://github.com/rabbitmq/ra
There’s an Elixir project that I’ve not tried yet but it’s a long this same vein. It does the registry part, but not sure if it currently does the load balancing part.
https://github.com/eliasdarruda/ra-registry