hubertlepicki

hubertlepicki

Long running Oban cron jobs

I have a system that needs to check for work every 5 minutes, and then process the payload. Processing the payload can take significantly more than 5 minutes.

Currently, Oban would just schedule extra jobs, that will be executed later. I have set the concurrency on the queue to 1, meaning no two jobs of the type execute at the same time. But the system is still doing a lot of unnecessary work after each long running job completes.

Is there a setting / pattern I can use to avoid scheduling additional jobs, while the Cron job is running? //cc @sorentwo

Most Liked

sorentwo

sorentwo

Oban Core Team

@hubertlepicki You can manage long-running cron jobs without overlap with a small tweak to the job’s unique settings. Here’s a complete example of a cron that is scheduled to run every minute but purposefully takes 65s to complete:

defmodule SlowCron do
  use Oban.Worker,
    max_attempts: 1,
    unique: [period: 300, states: ~w(available executing)a]

  def perform(job) do
    IO.puts("Started #{job.id}, inserted at #{job.inserted_at}")

    Process.sleep(:timer.seconds(65))

    IO.puts("Finished #{job.id}")

    :ok
  end
end

Oban.Test.Repo.start_link()

Oban.start_link(
  repo: Oban.Test.Repo,
  queues: [default: 10],
  plugins: [{Oban.Plugins.Cron, crontab: [{"* * * * *", SlowCron}]}]
)

The console output shows that the job ids are sequential but skip over the overlapping minute:

Started 16144, inserted at 2022-11-21 16:01:00.409060Z
Finished 16144
Started 16145, inserted at 2022-11-21 16:03:00.429351Z
Finished 16145
Started 16146, inserted at 2022-11-21 16:05:00.447115Z

The important part is extending the unique period while overriding the unique states so they don’t include completed jobs. This is safe because cron will only run on a single node in your cluster (the leader, according to Oban.Peer.leader?/0) regardless of the unique settings.

hubertlepicki

hubertlepicki

that was 50% why I asked it. It’s not obvious from the documentation. I could have experimented with unique: options but that’d be wasting some precious time I could spend playing with my kiddo. So I asked on Elixirforum and tagged @sorentwo . Hope that’s fine. Now it’s documented for everyone who can google.

hubertlepicki

hubertlepicki

@sorentwo that works thank you so much!

@dimitarvp we’re paying @sorentwo precisely not to have deal with all that xD.

Where Next?

Popular in Questions Top

nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
yawaramin
In the Dialyzer docs ( dialyzer — OTP 29.0.2 (dialyzer 6.0.1) ), there is a way to turn off a specific warning for a function: -dialyzer...
New

Other popular topics Top

mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement