Calculating the age of an oban job

Special Sections Oban
, ,
1

I’m trying to put together some alerts for the age of Oban jobs. I’d like to calculate the age of the job when it finishes by looking at the (attempted_at + duration) - first_scheduled_at time. I thought about using inserted_at in place of first_scheduled_at , but that doesn’t yield the proper calculation if it was a scheduled job.

Would this be the way to go about calculating job age and would you be up for adding db fields to be able to do observability around the age of a job?

(reposting this from elixir-slack)

1 Like
2

Thanks for reposting your question here.

Replicating my answers from Slack:

  1. We’re very conservative about adding new columns and favor using meta for extra information like this.
  2. If you want something consistent right away, then stash a timestamp in meta when you insert the job and use that for your calculation.

The issue with scheduled_at is that snoozing or retrying overwrites it. So, thinking about this a little more after our earlier, I realized that Oban can preserve the original timestamp itself (at least Pro can). This commit just landed in the v1.4 branch:

Record orig scheduling time on snooze and retry

Snoozing and rescheduling a job on error overwrites the scheduled_at
timestamp, which makes it impossible to determine how long a job was
truly waited before execution.

Now snoozing or erroring will inject the original scheduled_at
timestamp as unix seconds into the job’s meta as orig_scheduled_at.

You’ll be able to get a reliable scheduled_at for the age calculation:

case job.meta do
  %{"orig_scheduled_at" => unix_at} -> DateTime.from_unix(unix_at)
  _ -> job.scheduled_at
end
4 Likes
3

Record orig scheduling time on snooze and retry

That’s exactly what I was hoping for! :grin: Thank you so much!

Will I need to update my workers need to use Oban.Pro.Worker or will using Oban.Pro.Engines.Smart be enough?

4

Using the Smart engine is enough.

But, since you mentioned it, there aren’t any downsides to using Oban.Pro.Worker and it enables a lot.

5

Heads up, this change is out in Oban Pro v1.4.8 :slightly_smiling_face:

2 Likes
6

Thank you for getting it out so quickly!

One more request… job.scheduled_at and the rest of the timestamps have microsecond precision but job.meta["orig_scheduled_at"] only has second precision. I hate to ask, but what do you think about introducing a version of orig_scheduled_at with microsecond precision so I can do date math without needing to truncate to seconds.

Sorry I didn’t realize this earlier.

7

Excellent point. The version of orig_scheduled_at used by workflows is in seconds because that’s the granularity of scheduled jobs. They’re only staged to run once per second, so the precision doesn’t matter beyond that.

Both implementations would have to be switched to use microseconds instead as they share a common key.