dimitarvp
We have jobs with the following requirements:
- Jobs need to be retried exactly once every 24h or slightly less, with no exceptions – I am still working on the logic of another service (that we also control) returning tentative successes / failures because it has its own retry mechanism. We will capture the actual error response and do an Oban retry only then; in all other cases our other service either reports success or a tentative error – the latter means that it will use its own retry mechanism to try and do the job and only return a true error later. In the meantime: is snoozing the job the right tool in this case?
- Retries must always happen on schedule and we can’t afford to miss the window due to some extra time spent somewhere. See the remark below about a slightly tweaked
backofffor my stab at it. - Jobs should be attempted maximum 7 times.
- We don’t need a varying backoff; when it’s time to retry, the job should be retried immediately with no extra waiting. Hence, the
backoffis a flat value (24h or slightly sless). - We should not allow more than 50 parallel jobs running per second to not overwhelm the other service.
- Jobs
argsare a string-keyed map, which makes theunique’s:keysoption not applicable. I’ll rewrite the code to make the maps atom-keyed.
Here’s what I have in config/config.exs:
config :something, Oban,
repo: Something.Repo,
queues: [
some_queue: [limit: 50, dispatch_cooldown: :timer.seconds(1)]
]
and in the worker:
defmodule Something.Worker do
use Oban.Worker,
queue: :some_queue,
unique: [
period: :infinity,
states: [:available, :scheduled, :executing],
keys: [:user_id]
]
@timeout :timer.seconds(60)
@backoff 86400
@impl Oban.Worker
def timeout(_job), do: @timeout
@impl Worker
def backoff(%Job{attempt: _attempt}), do: @backoff
# `perform` and others follow.
end
and the enqueueing:
# This is not a compile-time value, the code is omitted for brevity.
Something.Worker.new(args, max_attempts: 7)
One thing I am not super clear on: is it possible to miss the retry window and so the next retry becomes effectively ~48h in the future?
I also thought of making sure I won’t miss the window of retrying via defensively deducting the timeout and some extra time, like so:
def backoff(%Job{attempt: _attempt}) do
# If we set a very low backoff value then pick the timeout value
# to avoid negative or too small a retry interval.
Enum.max([
@backoff - @timeout - 10,
@timeout
])
end
Questions:
- Is snoozing the job the right tool for when we have a tentative failure in our other service but we know it will keep retrying and so we can request information from it later? This also touches on analytics because we’d like to know how quickly and reliably do the jobs get executed (so we can also modify / improve the other service and/or tweak our timing values). EDIT: Seems like snoozing actually increases
attemptso it might not be what we’re looking for.
- Maybe I should just lower the retry period of the other service to f.ex. 23h55m so I don’t have to worry about missing retry windows? This too is in my control and started sounding like the better idea the more I was progressing with writing this post.
- Can we use string-keyed maps for
argsand enforce uniqueness on them?
Apropos, am I doing this whole thing right? Am I missing something?
Trending in Questions
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
Hello,
I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
Documentation
While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
So my question is quite simple and i have found no conclusive answer on forum, google or AI.
Should we use :erlang.float for Integer to ...
New
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New
Other Trending Topics
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
I am happy to introduce the very α version of the new programming language compiled to BEAM.
Welcome Cure.
It has literally three kille...
New
Hi there! We created Gust: A task orchestrator inspired by Airflow.
For those who have never heard about Aiflow, it’s a Python-based wor...
New
Hi everyone!
The first release candidate for the Expert language server project is now available!
We’ve published a press release detai...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security










Marked As Solved- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
sorentwo
Addressing your questions as best I can (in roughly the order they were asked):
Remember, the queue limit is per-node. If you’re running multiple nodes, or you’re doing rolling deploys, then the total number of allowed jobs will spike to 50*N, where N is the number of nodes.
The cooldown is used internally to prevent the queue’s producer from fetching jobs too rapidly. The default is 5ms. After jobs finish it will wait up to that amount of time before fetching more. This is the first I’ve seen anybody tweak that value, and bumping it to 1s could significantly slow down your processing if you have a backlog.
If the app is down, the queue is paused, or the backlog is deep enough the job may not run within that window. Guaranteeing that the job can run on schedule is up to your system.
Snoozing is a great approach for that, and exactly what we’d suggest you reach for.
Snoozing itself doesn’t increase the attempts, that’s from fetching the job in the first place. There’s a section in the Oban.Worker docs that covers how to compensate for the difference with backoffs. Pro’s Smart engine actually rolls back snooze attempts and records how many times a job has snoozed, FWIW.
Absolutely. The
argsmap is always converted to string keys, because it’s stored as JSON in the database. Thekeysare always converted to strings for unique queries as well.Also Liked
sorentwo
It is a demarcation line. Rate limiting in Pro is implemented at the engine level, so there isn’t any additional churn from fetching, snoozing, and retrying jobs that can’t really be ran. Rolling your own or using a library won’t be as efficient, but of course you can build whatever you like to on top of Oban
.
dimitarvp
Thank you.
dimitarvp
For future readers: it is not possible to specify string keys in the unique fields configuration or you’d get
** (ArgumentError) invalid value for :unique, expected :keys to be a list of atoms.However, specifying the keys as atoms works fine. Just wrote a few tests where I am trying to insert duplicate jobs and Oban correctly ignores the duplicates.
Last Post!
dimitarvp
Ohhh, nice! I am going to
stealtake inspiration from that.