leoschweizer
Hi,
we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume only jobs they can execute.
However, all models on a node share one GPU, requiring a concurrency limit of inference jobs per node across all model queues (e.g. one concurrent job only), while allowing different nodes to run concurrently.
Is there a supported way to enforce this at job-fetch time? Alternatively, can a single queue filter eligible jobs by node capabilities before fetching?
A worker-level lock either blocks jobs already marked executing or requires repeated snoozing. We’d prefer admission control that leaves jobs available until a compatible node has capacity.
What would be the recommended approach for such a scenario?
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #elixirconf-us
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance










Showing Posts 1 to 5- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
sorentwo
If I understand your need properly, you’re looking to batch related work together but limit per-node concurrency. That could be a good fit for setting a low per-node limit (
local_limit: 1), and then using a Chunk worker that partitions by args.Each chunk can group around a model argument, or some proxy for the model, like this:
That assumes any node can serve any model. If you want to filter by node capabilities, do that with the only option on Oban.Pro.Queue:
That way the
gpu_visionqueue only runs on vision capable nodes, etc.Hope that helps! Happy to provide more guidance if that was too abstract
leoschweizer
I think that doesn’t solve the issue. The core problem is that we have conflicting requirements regarding Oban’s intended usage (as far as we currently understand). Especially note that in the architecture as we currently envision it, nodes would each only run a subset of models and NOT each model.
Let me explain it differently:
Does that make sense?
So what we would e.g. need, if we wanted to go the route of a single queue, is a hook or mechanism that would allow us to filter eligible jobs at fetch time; thus we could control which jobs can run on a node while still maintaining concurrency control.
sorentwo
Yes, that makes sense, and now I understand your situation a bit better. You’re essentially looking for “node affinity”, where jobs are restricted to a certain node based on properties.
How complex is your topology? Meaning, how many different types of nodes are you running? It sounds like some nodes need to run queues for multiple models. Rather than routing jobs to a particular queue by model, you could route by GPU capability and retain the strict concurrency.
If putting related jobs in the same queue is impractical, you can still control overall concurrency on a node programmatically. Either by selectively
pausingandresuming, or actually scaling them up and down.There could be a Pro feature that this all points to, but I’m hoping we can figure out a solution that works for you currently.
leoschweizer
Well, we currently are still in a prototyping phase, so that means that
a) the topology we would run this on is very diverse, i.e. running this on existing office infrastructure where (hypothetical example) one node would be serving a classification and a transcription model and another node would be serving a reasoning model (but none of the others). If we went with a snigle queue (which is still our overall preferred approach), we could not prevent classification jobs from being routed to the reasoning node and vice versa.
and b) we have no pressing urgency (yet) to implement that right away. If a (near) future Oban (Pro) release would offer a mechanism to tackle that “cleanly” we can postpone the implementation and skip the workaround.
The workaround we currently came up with would be to use one queue per model (for managing the node-affinity), and on each node establish a shared lock between all GPU workers outside of Oban that would make non-lock-holders repeatedly snooze their tasks until they can acquire their lock (for managing the concurrency). But I think that would introduce other issues, like blocking a task on a node, when it could potentially already run on another idle node.
I don’t really understand what you mean by that, or how you imagine it would map to Oban concepts; could you please expand on that a little?
sorentwo
You shouldn’t need to use a lock, that’s really what the queue is for. The queue is the arbiter of what to run concurrently. The trick is simply to put constrained jobs in the same queue(s).
I mean if you have the same class of GPU for two models, and you want to restrict concurrent access to that GPU resource, put those jobs in the same queue rather than splitting them into different queue by model.