leoschweizer

leoschweizer

Hi,

we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume only jobs they can execute.

However, all models on a node share one GPU, requiring a concurrency limit of inference jobs per node across all model queues (e.g. one concurrent job only), while allowing different nodes to run concurrently.

Is there a supported way to enforce this at job-fetch time? Alternatively, can a single queue filter eligible jobs by node capabilities before fetching?

A worker-level lock either blocks jobs already marked executing or requires repeated snoozing. We’d prefer admission control that leaves jobs available until a compatible node has capacity.

What would be the recommended approach for such a scenario?

Showing Posts 1 to 5

sorentwo

sorentwo

Oban Core Team

If I understand your need properly, you’re looking to batch related work together but limit per-node concurrency. That could be a good fit for setting a low per-node limit (local_limit: 1), and then using a Chunk worker that partitions by args.

Each chunk can group around a model argument, or some proxy for the model, like this:

defmodule MyApp.InferenceWorker do
  use Oban.Pro.Chunk, queue: :inference, by: [args: :model], size: 32, timeout: 1_000

  args_schema do
    field :model, :enum, values: ~w(list of possible modules)a
  end

  @impl true
  def process([%{args: %{model: model}} | _] = jobs) do
    # run one batched inference for `model` over all jobs
  end
end

That assumes any node can serve any model. If you want to filter by node capabilities, do that with the only option on Oban.Pro.Queue:

queues: [
  gpu_vision: [local_limit: 1, only: {:sys_env, "GPU_POOL", "vision"}],
  gpu_text:   [local_limit: 1, only: {:sys_env, "GPU_POOL", "text"}]
]

That way the gpu_vision queue only runs on vision capable nodes, etc.

Hope that helps! Happy to provide more guidance if that was too abstract :slightly_smiling_face:

leoschweizer

leoschweizer OP

I think that doesn’t solve the issue. The core problem is that we have conflicting requirements regarding Oban’s intended usage (as far as we currently understand). Especially note that in the architecture as we currently envision it, nodes would each only run a subset of models and NOT each model.

Let me explain it differently:

  • We have a constrained ressource for which we require strict concurrency control - the GPU. The obvious choice in Oban terms would be to have a single queue for all jobs / workers requiring access to this ressource, which would give us that exact concurrency control.
  • But we also have constrained capabilities per node in the cluster, such that not every node can execute every job; in Oban terms, the solution to that would be to have different queues for different jobs, which gives us the control about what to execute on which node.
  • If we do that, we lose the concurrency control again - but we kind of need both.

Does that make sense?

So what we would e.g. need, if we wanted to go the route of a single queue, is a hook or mechanism that would allow us to filter eligible jobs at fetch time; thus we could control which jobs can run on a node while still maintaining concurrency control.

sorentwo

sorentwo

Oban Core Team

Yes, that makes sense, and now I understand your situation a bit better. You’re essentially looking for “node affinity”, where jobs are restricted to a certain node based on properties.

How complex is your topology? Meaning, how many different types of nodes are you running? It sounds like some nodes need to run queues for multiple models. Rather than routing jobs to a particular queue by model, you could route by GPU capability and retain the strict concurrency.

If putting related jobs in the same queue is impractical, you can still control overall concurrency on a node programmatically. Either by selectively pausing and resuming, or actually scaling them up and down.

There could be a Pro feature that this all points to, but I’m hoping we can figure out a solution that works for you currently.

leoschweizer

leoschweizer OP

Well, we currently are still in a prototyping phase, so that means that

a) the topology we would run this on is very diverse, i.e. running this on existing office infrastructure where (hypothetical example) one node would be serving a classification and a transcription model and another node would be serving a reasoning model (but none of the others). If we went with a snigle queue (which is still our overall preferred approach), we could not prevent classification jobs from being routed to the reasoning node and vice versa.

and b) we have no pressing urgency (yet) to implement that right away. If a (near) future Oban (Pro) release would offer a mechanism to tackle that “cleanly” we can postpone the implementation and skip the workaround.

The workaround we currently came up with would be to use one queue per model (for managing the node-affinity), and on each node establish a shared lock between all GPU workers outside of Oban that would make non-lock-holders repeatedly snooze their tasks until they can acquire their lock (for managing the concurrency). But I think that would introduce other issues, like blocking a task on a node, when it could potentially already run on another idle node.

Rather than routing jobs to a particular queue by model, you could route by GPU capability and retain the strict concurrency.

I don’t really understand what you mean by that, or how you imagine it would map to Oban concepts; could you please expand on that a little?

sorentwo

sorentwo

Oban Core Team

You shouldn’t need to use a lock, that’s really what the queue is for. The queue is the arbiter of what to run concurrently. The trick is simply to put constrained jobs in the same queue(s).

I mean if you have the same class of GPU for two models, and you want to restrict concurrent access to that GPU resource, put those jobs in the same queue rather than splitting them into different queue by model.

— All posts loaded —

Where Next? Top

Trending in Questions Top

stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New
itzmidinesh
Hi all, I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
leoschweizer
Hi, we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume ...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews