udoschneider
Distributed Registry/Service Discovery/Worker pools in non-uniform clusters
I’m struggling to understand a problem I’m facing, and I’m not sure if there’s an existing solution.
In the project I’m working on, we need to create embeddings using Bumblebee in conjunction with pgvector. Currently, all nodes in the cluster are running the same release. However, with the new requirements for embeddings generation, I want some nodes to have more powerful GPUs. These nodes will detect the presence of a GPU using an environment variable and will start the Bumblebee Serving.
The core idea is that all nodes can request embeddings generation, but the actual worker processes will only run on the GPU nodes.
I’m looking for a solution that combines a distributed registry with something like Poolboy, which would allow for round-robin or other load balancing on the registered processes. I’ve looked into HordeRegistry, but it seems to be designed for single processes only.
I have some ideas for building a custom solution from scratch, but I have a feeling that my requirements—such as service discovery and load distribution—might be met with an off-the-shelf solution.
Do you have any pointers?
Thanks,
Udo
Most Liked
hubertlepicki
sir, please do not encourage me. I ended up discussing medieval literature and Don Quixote in particular the last time. On this forum.
garrison
As much as I love fault-tolerant consensus (and you know I do), I don’t think this is really necessary here. If you want exactly-once semantics on the queue you would have to store all of your data in Ra (which will certainly never scale), and process registries are a use-case where I’m not entirely convinced you actually want strong consistency guarantees - because where you really want those guarantees is on the database the processes are writing to. Once the database guarantees correctness the process registry and such only needs good performance (it should be right most of the time), so you don’t really need consensus anymore.
(I have had the misfortune of thinking about this a lot lately.)
But here the data is already stored in Postgres! So if you just put your work queue in Postgres you get transactions over both and you are totally safe correctness-wise (read committed footguns notwithstanding).
Like you could literally just do this:
def insert_data(text) do
%Data{id: id} = Repo.insert!(%Data{text: text})
Repo.insert! %EmbedJob{data_id: id, lease: nil}
end
def pop_job do
expired = DateTime.utc_now() |> DateTime.add(-1, :minute)
Repo.transaction(fn ->
job = Repo.one!(from j in EmbedJob,
where: is_nil(j.lease) or j.lease < ^expired,
order_by: :inserted_at, limit: 1, lock: "for update", preload: :data)
Repo.update! change(job, %{lease: DateTime.utc_now()})
end)
end
def worker do
{:ok, job} = pop_job()
vector = embed_text(job.data.text)
Repo.transaction(fn ->
%Job{} = Repo.one!(from j in EmbedJob, where: j.id == ^job.id, lock: "for update")
Repo.update! change(job.data, %{vector: vector})
Repo.delete! job
end)
end
And now you don’t need a process registry at all!
hubertlepicki
I’d default to Oban, and run some queues only on selected nodes.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









