dimitarvp

dimitarvp

We have jobs with the following requirements:

  • Jobs need to be retried exactly once every 24h or slightly less, with no exceptions – I am still working on the logic of another service (that we also control) returning tentative successes / failures because it has its own retry mechanism. We will capture the actual error response and do an Oban retry only then; in all other cases our other service either reports success or a tentative error – the latter means that it will use its own retry mechanism to try and do the job and only return a true error later. In the meantime: is snoozing the job the right tool in this case?
  • Retries must always happen on schedule and we can’t afford to miss the window due to some extra time spent somewhere. See the remark below about a slightly tweaked backoff for my stab at it.
  • Jobs should be attempted maximum 7 times.
  • We don’t need a varying backoff; when it’s time to retry, the job should be retried immediately with no extra waiting. Hence, the backoff is a flat value (24h or slightly sless).
  • We should not allow more than 50 parallel jobs running per second to not overwhelm the other service.
  • Jobs args are a string-keyed map, which makes the unique’s :keys option not applicable. I’ll rewrite the code to make the maps atom-keyed.

Here’s what I have in config/config.exs:

config :something, Oban,
  repo: Something.Repo,
  queues: [
    some_queue: [limit: 50, dispatch_cooldown: :timer.seconds(1)]
  ]

and in the worker:

defmodule Something.Worker do
  use Oban.Worker,
    queue: :some_queue,
    unique: [
      period: :infinity,
      states: [:available, :scheduled, :executing],
      keys: [:user_id]
    ]

  @timeout :timer.seconds(60)
  @backoff 86400

  @impl Oban.Worker
  def timeout(_job), do: @timeout

  @impl Worker
  def backoff(%Job{attempt: _attempt}), do: @backoff

  # `perform` and others follow.
end

and the enqueueing:

# This is not a compile-time value, the code is omitted for brevity.
Something.Worker.new(args, max_attempts: 7)

One thing I am not super clear on: is it possible to miss the retry window and so the next retry becomes effectively ~48h in the future?

I also thought of making sure I won’t miss the window of retrying via defensively deducting the timeout and some extra time, like so:

  def backoff(%Job{attempt: _attempt}) do
    # If we set a very low backoff value then pick the timeout value
    # to avoid negative or too small a retry interval.
    Enum.max([
      @backoff - @timeout - 10,
      @timeout
    ])
  end

Questions:

  • Is snoozing the job the right tool for when we have a tentative failure in our other service but we know it will keep retrying and so we can request information from it later? This also touches on analytics because we’d like to know how quickly and reliably do the jobs get executed (so we can also modify / improve the other service and/or tweak our timing values). EDIT: Seems like snoozing actually increases attempt so it might not be what we’re looking for. :confused:
  • Maybe I should just lower the retry period of the other service to f.ex. 23h55m so I don’t have to worry about missing retry windows? This too is in my control and started sounding like the better idea the more I was progressing with writing this post.
  • Can we use string-keyed maps for args and enforce uniqueness on them?

Apropos, am I doing this whole thing right? Am I missing something?

Marked As Solved

sorentwo

sorentwo

Oban Core Team

Addressing your questions as best I can (in roughly the order they were asked):

Remember, the queue limit is per-node. If you’re running multiple nodes, or you’re doing rolling deploys, then the total number of allowed jobs will spike to 50*N, where N is the number of nodes.

The cooldown is used internally to prevent the queue’s producer from fetching jobs too rapidly. The default is 5ms. After jobs finish it will wait up to that amount of time before fetching more. This is the first I’ve seen anybody tweak that value, and bumping it to 1s could significantly slow down your processing if you have a backlog.

If the app is down, the queue is paused, or the backlog is deep enough the job may not run within that window. Guaranteeing that the job can run on schedule is up to your system.

Snoozing is a great approach for that, and exactly what we’d suggest you reach for.

Snoozing itself doesn’t increase the attempts, that’s from fetching the job in the first place. There’s a section in the Oban.Worker docs that covers how to compensate for the difference with backoffs. Pro’s Smart engine actually rolls back snooze attempts and records how many times a job has snoozed, FWIW.

Absolutely. The args map is always converted to string keys, because it’s stored as JSON in the database. The keys are always converted to strings for unique queries as well.

Also Liked

sorentwo

sorentwo

Oban Core Team

It is a demarcation line. Rate limiting in Pro is implemented at the engine level, so there isn’t any additional churn from fetching, snoozing, and retrying jobs that can’t really be ran. Rolling your own or using a library won’t be as efficient, but of course you can build whatever you like to on top of Oban :slightly_smiling_face:.

dimitarvp

dimitarvp OP

Thank you.

dimitarvp

dimitarvp OP

For future readers: it is not possible to specify string keys in the unique fields configuration or you’d get ** (ArgumentError) invalid value for :unique, expected :keys to be a list of atoms.

However, specifying the keys as atoms works fine. Just wrote a few tests where I am trying to insert duplicate jobs and Oban correctly ignores the duplicates.

Last Post!

dimitarvp

dimitarvp OP

Ohhh, nice! I am going to steal take inspiration from that. :smiley:

Where Next? Top

Trending in Questions Top

RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
nseaSeb
Hello, I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
samoloth
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
asweet-confluent
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews