We have jobs with the following requirements:

• Jobs need to be retried exactly once every 24h or slightly less, with no exceptions – I am still working on the logic of another service (that we also control) returning tentative successes / failures because it has its own retry mechanism. We will capture the actual error response and do an Oban retry only then; in all other cases our other service either reports success or a tentative error – the latter means that it will use its own retry mechanism to try and do the job and only return a true error later. In the meantime: is snoozing the job the right tool in this case?
• Retries must always happen on schedule and we can’t afford to miss the window due to some extra time spent somewhere. See the remark below about a slightly tweaked backoff for my stab at it.
• Jobs should be attempted maximum 7 times.
• We don’t need a varying backoff; when it’s time to retry, the job should be retried immediately with no extra waiting. Hence, the backoff is a flat value (24h or slightly sless).
• We should not allow more than 50 parallel jobs running per second to not overwhelm the other service.
• Jobs args are a string-keyed map, which makes the unique’s :keys option not applicable. I’ll rewrite the code to make the maps atom-keyed.

Here’s what I have in config/config.exs:

config :something, Oban,
  repo: Something.Repo,
  queues: [
    some_queue: [limit: 50, dispatch_cooldown: :timer.seconds(1)]
  ]

and in the worker:

defmodule Something.Worker do
  use Oban.Worker,
    queue: :some_queue,
    unique: [
      period: :infinity,
      states: [:available, :scheduled, :executing],
      keys: [:user_id]
    ]

  @timeout :timer.seconds(60)
  @backoff 86400

  @impl Oban.Worker
  def timeout(_job), do: @timeout

  @impl Worker
  def backoff(%Job{attempt: _attempt}), do: @backoff

  # `perform` and others follow.
end

and the enqueueing:

# This is not a compile-time value, the code is omitted for brevity.
Something.Worker.new(args, max_attempts: 7)

One thing I am not super clear on: is it possible to miss the retry window and so the next retry becomes effectively ~48h in the future?

I also thought of making sure I won’t miss the window of retrying via defensively deducting the timeout and some extra time, like so:

  def backoff(%Job{attempt: _attempt}) do
    # If we set a very low backoff value then pick the timeout value
    # to avoid negative or too small a retry interval.
    Enum.max([
      @backoff - @timeout - 10,
      @timeout
    ])
  end

Questions:

• Is snoozing the job the right tool for when we have a tentative failure in our other service but we know it will keep retrying and so we can request information from it later? This also touches on analytics because we’d like to know how quickly and reliably do the jobs get executed (so we can also modify / improve the other service and/or tweak our timing values). EDIT: Seems like snoozing actually increases attempt so it might not be what we’re looking for.
• Maybe I should just lower the retry period of the other service to f.ex. 23h55m so I don’t have to worry about missing retry windows? This too is in my control and started sounding like the better idea the more I was progressing with writing this post.
• Can we use string-keyed maps for args and enforce uniqueness on them?

Apropos, am I doing this whole thing right? Am I missing something?