Unique Periodic Worker, no overlaps when executing for longer than interval

Special Sections Oban
,
1

I want an Oban worker which runs periodically (e.g. every hour), but should be unique, so if the worker takes longer than the period between runs another worker is not enqueued.

I have an Elixir cluster with Oban Pro.

I’m using the Cron plug configured as such:

config :shared, Oban,
  engine: Oban.Pro.Engines.Smart,
  queues: [exports: 1],
  repo: Shared.Infrastructure.Repo,
  plugins: [
    {
      Oban.Plugins.Cron,
      crontab: [
        # every min for testing purposes
        {"* * * * *", Export.Standard.Worker}
      ]
    }
  ]

My first native approach was this:

use Oban.Pro.Worker,
  queue: :exports,
  unique: :infinity

However this means the worker only ever runs once since it accepts no arguments, so the unique fields, worker, queue and arg’s, are always the same, and given the time period is infinity it never runs again.

So I tried adding in the states:

use Oban.Pro.Worker,
  queue: :exports,
  unique: [period: :infinity, states: [:available, :executing]]

Then I tried removing the unique and setting global_limit for the queue to 1, this means only one worker can run at once.

config :shared, Oban,
  engine: Oban.Pro.Engines.Smart,
  queues: [
    exports: [global_limit: 1]
  ],
  repo: Shared.Infrastructure.Repo,
  plugins: [
    {
      Oban.Plugins.Cron,
      crontab: [
        # every min for testing purposes
        {"* * * * *", Export.Standard.Worker}
      ]
    }
  ]

use Oban.Pro.Worker,
  queue: :exports

This seems to work well enough.

But I do notice however that I accumulate a lot of jobs in “available” state, I think this is because cron starts the worker but can’t run it because of the global_limit of 1. So they build up indefinitely.

Is there anything I’m missing or a better approach to this perhaps?

2

What was the problem with this one? Looks good enough to me.

3

Hi Pro customer! :star2:

If it takes longer for the job to run than the period.. and you serialize them, they’re going to build up indefinitely.

Here are some options to improve your approach:

  • Make the jobs run faster
  • Queue them farther apart
  • Shed load by not enqueuing jobs while the other jobs are waiting.(Which you would accomplish with uniqueness.)
3 Likes
4

If I kill the node while a job is executing it leaves the job in an executing state.

When a new node is started it never run the worker again, I assume this is because there is already an executing job, which matches the unique state, it just doesn’t know that the job is orphaned.

But what I’ve realised I can add the LifeLine plugin.

This makes the “executing” job “available” after 10 mins.

1 Like
5

Absolutely a necessity for running in production (as noted in the Ready for Production guide). Since you have Pro, you should use the DynamicLifeline instead though. Switching out the Lifeline is also mentioned in Pro’s Adoption guide too :slightly_smiling_face:

4 Likes
6

My final solution which seems to work well is:

  use Oban.Pro.Worker,
    queue: :exports,
    tags: ["standard"],
    unique: [period: :infinity, states: [:scheduled, :available, :executing], timestamp: :scheduled_at]

And to limit the queue size to 1 globally with DynamicLifeLine to restart any orphaned jobs.

config :shared, Oban,
  engine: Oban.Pro.Engines.Smart,
  queues: [
    default: 10,
    exports: [global_limit: 1],
  ],
  repo: Shared.Infrastructure.Repo,
  plugins: [
    {
      Oban.Plugins.Cron,
      crontab: [
        # every min for testing purposes
        {"* * * * *", Export.Standard.Worker}
      ]
    },
    {Oban.Pro.Plugins.DynamicLifeline, rescue_interval: :timer.minutes(10)},
    # ...  ]

Thanks for the help.

P.S I feel like there is so much that can be done with Oban Pro it would be worth some premium content to demonstrate different patterns and configurations.

4 Likes
7

We are currently working on our Pro Training modules!

Won’t be long! :surfing_woman:

2 Likes