hubertlepicki
Long running Oban cron jobs
I have a system that needs to check for work every 5 minutes, and then process the payload. Processing the payload can take significantly more than 5 minutes.
Currently, Oban would just schedule extra jobs, that will be executed later. I have set the concurrency on the queue to 1, meaning no two jobs of the type execute at the same time. But the system is still doing a lot of unnecessary work after each long running job completes.
Is there a setting / pattern I can use to avoid scheduling additional jobs, while the Cron job is running? //cc @sorentwo
Most Liked
sorentwo
@hubertlepicki You can manage long-running cron jobs without overlap with a small tweak to the job’s unique settings. Here’s a complete example of a cron that is scheduled to run every minute but purposefully takes 65s to complete:
defmodule SlowCron do
use Oban.Worker,
max_attempts: 1,
unique: [period: 300, states: ~w(available executing)a]
def perform(job) do
IO.puts("Started #{job.id}, inserted at #{job.inserted_at}")
Process.sleep(:timer.seconds(65))
IO.puts("Finished #{job.id}")
:ok
end
end
Oban.Test.Repo.start_link()
Oban.start_link(
repo: Oban.Test.Repo,
queues: [default: 10],
plugins: [{Oban.Plugins.Cron, crontab: [{"* * * * *", SlowCron}]}]
)
The console output shows that the job ids are sequential but skip over the overlapping minute:
Started 16144, inserted at 2022-11-21 16:01:00.409060Z
Finished 16144
Started 16145, inserted at 2022-11-21 16:03:00.429351Z
Finished 16145
Started 16146, inserted at 2022-11-21 16:05:00.447115Z
The important part is extending the unique period while overriding the unique states so they don’t include completed jobs. This is safe because cron will only run on a single node in your cluster (the leader, according to Oban.Peer.leader?/0) regardless of the unique settings.
hubertlepicki
that was 50% why I asked it. It’s not obvious from the documentation. I could have experimented with unique: options but that’d be wasting some precious time I could spend playing with my kiddo. So I asked on Elixirforum and tagged @sorentwo . Hope that’s fine. Now it’s documented for everyone who can google.
hubertlepicki
@sorentwo that works thank you so much!
@dimitarvp we’re paying @sorentwo precisely not to have deal with all that xD.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









