How to make Oban run more frequently than once in a minute?

Questions / Help
, ,
1

I’m using Oban for the ongoing tasks.

# .....
    {Oban.Plugins.Cron,
     crontab: [
       {"* * * * *", MyApp.Workers.W1},
       {"* * * * *", MyApp.Workers.W2},
     ]},

I now need to run W1 and W2 more frequently that every minute - around once in every 10…30 seconds. Since cron doesn’t support higher frequency than 1/min, how would I get around this limitation? Preferably without hacks, unless absolutely necessary.

I don’t consider switching from Oban to other library.

I might resort to GenServer itself, though.

2

Yeah I think this is the right call, can you elaborate a bit on your use case?

3

Regular tasks - scan Db, fetch items that have a certain status, process them, update status.

I also need to be able to a) see if there’ve been any errors b) re-execute a task

Unlike GenServer, Oban supports both of these. While the the 1st one can be achived via logging an error via Logger, re-executing a failed task isn’t trivial. Or is it? How would I do it in GenServer in a simple manner, and with no additional external dependencies?

4

I think this is the same question you asked on StackOverflow: cron - How to make Oban run more frequently than once in a minute? - Stack Overflow

I answered there :upside_down_face:

5

Yes. I now have introduced 2 additional variables – error handling or logging, and re-execution.

6

I would consider trying to insert the job at the same time that status is set, instead of essentially “polling” the database.

If you must poll, then yeah I’d just do a genserver that queries every N seconds and then inserts Oban jobs. Make sure to set some unique settings for the job so that when you run multiple nodes you don’t get duplicates.

1 Like
7

As @benwilson512 suggested, I’d try to schedule the job at the same time a status changes. You can bundle the update and job insertion in the same transaction:

# Assuming you have a changeset and know when to check next
Ecto.Multi.new()
|> Ecto.Multi.update(:record, changeset)
|> Oban.insert(:check, Worker.new(%{id: record.id}, schedule_in: {30, :seconds}))

Failing that, if there are unknown factors you’re checking for, you can also insert a few sub-minute jobs at each Cron run:

# Set a scheduler arg in the crontab and a step to control seconds apart
{"* * * * *", MyWorker, args: %{scheduler: true, step: 10}}

# Match on that in the worker
def perform(%Job{args: %{"scheduler" => true, "step" => step}}) do
  1..60//step
  |> Enum.map(&new(%{}, schedule_in: &1))
  |> Oban.insert_all()

  :ok
end

Cron always runs at the top of the minute, so you’ll schedule jobs [1, 11, 21, 31, 41, 51] seconds out.