tedi

tedi

Unique job handling

Hello,

I am using Oban [Pro] for all of my queues.

I have been hitting a wall on one of my queue functionalities. I have the following conditions:

  • Users update their data for a certain resource quite frequently. There are many instances where the user triggers an update action before a job completes. I have a unique clause by the keys user_id and resource_id
  • I need the job to execute one at a time when that uniqueness clause is true.
  • If a job with the user_id and resource_id is already running, I would like to queue the new job to run after the already running job is completed.
  • If this happens more than once and there’s already another job in the queue with the same keys, overwrite the one that’s queued to start next. Example:
    • Job 1 is executing
    • Job 2 with the same user_id and resource_id is queue to run next
    • We receive job 3 with the same user_id/resource_id. I need Job 2 to be replaced with job 3 (and job 4,5,6…etc do the same)

This is what I have:

      MyApp.Workers.MyJobWorker.new(
        args,
        queue: :test_queue,
        unique: [period: :infinity, fields: [:args], keys: [:user_id, :resource_id]],
        replace: [:args]
      )
      |> Oban.insert()

However with this set-up, the new jobs don’t get stored when there is a unique conflict.

Thanks,
Tedi

Marked As Solved

sorentwo

sorentwo

Oban Core Team

All you’re missing is setting the unique states. You only want to consider :available and :scheduled states for uniqueness (and maybe :retryable).

[
  period: :infinity,
  fields: [:args],
  keys: [:user_id, :resource_id],
  states: [:available, :scheduled]
]

Also Liked

tedi

tedi

Ah I see, that did it, thank you!

Just to confirm, for the config, I have the following set-up:

test_queue: [
          local_limit: 10,
          rate_limit: [allowed: 1, period: 30, partition: [fields: [:args], keys: [:user_id, :resource_id]]]
        ],

Seems to be fine from initial testing, but this would work globally across nodes as well, correct? Capping each job to run per user at 30 seconds.

sorentwo

sorentwo

Oban Core Team

That’s right. You’ve got it!

Last Post!

tedi

tedi

Awesome, thank you for your help!

Where Next?

Popular in Questions Top

Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

We're in Beta

About us Mission Statement