carterbryden

carterbryden

Oban: having many dynamic queues okay or a bad idea?

Hi, wondering if anyone has experience creating a lot of queues dynamically with oban, and how that went for you.

I’m writing an app that will be connecting over SSH to potentially N number of servers per user, and I want to create a queue for each server. The idea is to always wait for the last operation on that server to finish before beginning the next.

It seems like theoretically there shouldn’t be an issue for this with Oban, but has anyone tried it out? Any issues?

Also, should I store these queues in a db table so that I can start them up again easily later? Or has anyone done this differently?

Thanks!

Most Liked

sorentwo

sorentwo

Oban Core Team

Please pardon the slow reply, I’ve been on vacation for the past few weeks

This is entirely doable with a single worker through the use of :snooze. Here’s a pseudo example:

defmodule MyApp.SequentialWorker do
  use Oban.Worker, queue: :some_queue

  import Ecto.Query, only: [where: 3]

  @impl Worker
  def perform(%Job{args: args}) do
    if executable_for_user?(args) do
      # do normal stuff
    else
      # snooze for 60 seconds, make it as short as necessary
      {:snooze, 60}
    end
  end

  defp executable_for_user?(%{"user_id" => user_id}) do
    Job
    |> where(j, j.worker == Worker.to_string(__MODULE__))
    |> where(j, j.queue == "some_queue")
    |> where(j, j.state == "executing")
    |> where(j, fragment("? @> ?", j.args, ^%{user_id: user_id})
    |> MyApp.Repo.exists?()
  end
end

By checking whether there is another job executing you can force global concurrency. In this case the worker checks for other executing workers with the same user_id, though that is arbitrary and you could scope to anything you like.

It isn’t as optimized as running multiple queues with global concurrency, but it will work right now and is accurate.

sorentwo

sorentwo

Oban Core Team

Partitioned rate-limiting landed in Pro Log in | Hex. Thanks for all the support! :yellow_heart:

sorentwo

sorentwo

Oban Core Team

Working on it now that some major :ox::razor: is finished. Of course, there’ve been many changes behind the scenes to improve queue option validation and serialization, but the groundwork is there.

As soon as something is ready for testing, I’ll announce it here :+1:

Where Next?

Popular in Questions Top

chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 127089 1222
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement