dimitarvp

dimitarvp

We have jobs with the following requirements:

  • Jobs need to be retried exactly once every 24h or slightly less, with no exceptions – I am still working on the logic of another service (that we also control) returning tentative successes / failures because it has its own retry mechanism. We will capture the actual error response and do an Oban retry only then; in all other cases our other service either reports success or a tentative error – the latter means that it will use its own retry mechanism to try and do the job and only return a true error later. In the meantime: is snoozing the job the right tool in this case?
  • Retries must always happen on schedule and we can’t afford to miss the window due to some extra time spent somewhere. See the remark below about a slightly tweaked backoff for my stab at it.
  • Jobs should be attempted maximum 7 times.
  • We don’t need a varying backoff; when it’s time to retry, the job should be retried immediately with no extra waiting. Hence, the backoff is a flat value (24h or slightly sless).
  • We should not allow more than 50 parallel jobs running per second to not overwhelm the other service.
  • Jobs args are a string-keyed map, which makes the unique’s :keys option not applicable. I’ll rewrite the code to make the maps atom-keyed.

Here’s what I have in config/config.exs:

config :something, Oban,
  repo: Something.Repo,
  queues: [
    some_queue: [limit: 50, dispatch_cooldown: :timer.seconds(1)]
  ]

and in the worker:

defmodule Something.Worker do
  use Oban.Worker,
    queue: :some_queue,
    unique: [
      period: :infinity,
      states: [:available, :scheduled, :executing],
      keys: [:user_id]
    ]

  @timeout :timer.seconds(60)
  @backoff 86400

  @impl Oban.Worker
  def timeout(_job), do: @timeout

  @impl Worker
  def backoff(%Job{attempt: _attempt}), do: @backoff

  # `perform` and others follow.
end

and the enqueueing:

# This is not a compile-time value, the code is omitted for brevity.
Something.Worker.new(args, max_attempts: 7)

One thing I am not super clear on: is it possible to miss the retry window and so the next retry becomes effectively ~48h in the future?

I also thought of making sure I won’t miss the window of retrying via defensively deducting the timeout and some extra time, like so:

  def backoff(%Job{attempt: _attempt}) do
    # If we set a very low backoff value then pick the timeout value
    # to avoid negative or too small a retry interval.
    Enum.max([
      @backoff - @timeout - 10,
      @timeout
    ])
  end

Questions:

  • Is snoozing the job the right tool for when we have a tentative failure in our other service but we know it will keep retrying and so we can request information from it later? This also touches on analytics because we’d like to know how quickly and reliably do the jobs get executed (so we can also modify / improve the other service and/or tweak our timing values). EDIT: Seems like snoozing actually increases attempt so it might not be what we’re looking for. :confused:
  • Maybe I should just lower the retry period of the other service to f.ex. 23h55m so I don’t have to worry about missing retry windows? This too is in my control and started sounding like the better idea the more I was progressing with writing this post.
  • Can we use string-keyed maps for args and enforce uniqueness on them?

Apropos, am I doing this whole thing right? Am I missing something?

Showing Posts 1 to 8

dimitarvp

dimitarvp OP

Pinging @sorenone and @sorentwo as this is a work assignment and I haven’t used Oban in a while.

sorentwo

sorentwo

Oban Core Team

Addressing your questions as best I can (in roughly the order they were asked):

Remember, the queue limit is per-node. If you’re running multiple nodes, or you’re doing rolling deploys, then the total number of allowed jobs will spike to 50*N, where N is the number of nodes.

The cooldown is used internally to prevent the queue’s producer from fetching jobs too rapidly. The default is 5ms. After jobs finish it will wait up to that amount of time before fetching more. This is the first I’ve seen anybody tweak that value, and bumping it to 1s could significantly slow down your processing if you have a backlog.

If the app is down, the queue is paused, or the backlog is deep enough the job may not run within that window. Guaranteeing that the job can run on schedule is up to your system.

Snoozing is a great approach for that, and exactly what we’d suggest you reach for.

Snoozing itself doesn’t increase the attempts, that’s from fetching the job in the first place. There’s a section in the Oban.Worker docs that covers how to compensate for the difference with backoffs. Pro’s Smart engine actually rolls back snooze attempts and records how many times a job has snoozed, FWIW.

Absolutely. The args map is always converted to string keys, because it’s stored as JSON in the database. The keys are always converted to strings for unique queries as well.

dimitarvp

dimitarvp OP

Thanks a bunch, a lot has been cleared up. :heart: A few more questions and remarks if you don’t mind.

I am aware, for now we’re on a single node. Thanks for bringing it up for future readers. :+1:

Fair. What would you recommend for a firm limitation of 50 jobs / sec if we are not buying Oban Pro just yet? I mean sure, I can roll my own rate limiting – or use one of the several libraries to do so – I am just curious if the free version of Oban can be used to that effect, or is this one of the things that you put as a demarcation line for buying Pro?

Thought snoozing increases attempt but it actually increases max_attemps – and then actually executing the job increases attempt as it should. My bad for misreading the first part.

Fantastic. It seems my editor was screwing with me because when I specified unique: ["user_id"] it yelled at me that only atoms are allowed there. I’ll check again in more details.

sorentwo

sorentwo

Oban Core Team

It is a demarcation line. Rate limiting in Pro is implemented at the engine level, so there isn’t any additional churn from fetching, snoozing, and retrying jobs that can’t really be ran. Rolling your own or using a library won’t be as efficient, but of course you can build whatever you like to on top of Oban :slightly_smiling_face:.

dimitarvp

dimitarvp OP

Thank you.

dimitarvp

dimitarvp OP

For future readers: it is not possible to specify string keys in the unique fields configuration or you’d get ** (ArgumentError) invalid value for :unique, expected :keys to be a list of atoms.

However, specifying the keys as atoms works fine. Just wrote a few tests where I am trying to insert duplicate jobs and Oban correctly ignores the duplicates.

sorenone

sorenone

Oban Core Team

The Oban.Job typespec shows that keys expects a list of atoms, and that’s the only version shown in the various examples. Yep, the docs on uniqueness need some love and attention. :face_with_open_eyes_and_hand_over_mouth:

Indeed! That’s exactly how it’s done in the tests.

dimitarvp

dimitarvp OP

Ohhh, nice! I am going to steal take inspiration from that. :smiley:

— All posts loaded —

Where Next? Top

Trending in Questions Top

RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
samoloth
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
FlyingNoodle
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
New
ryanwinchester
apply_graft/2 doesn’t rewrite an add_many sub-workflow’s deps on an add step. Grafted jobs cancel with “upstream job was deleted” Version...
New

Other Trending Topics Top

mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews