Handling failed Oban jobs

Phoenix Forum Questions / Help
, , , , ,
1

Hey Devs,

I have been using Oban — Oban v2.5.0 (hexdocs.pm) in one of my phoenix application. My requirement is like when job fails after trying couple of times based on the max attempts that has been configured to that job. I need to update that job has been failed in DB and give an option to retry the job. I am able to accomplish this using Oban.Telemetry — Oban v2.5.0 (hexdocs.pm) events & Oban.retry. But is there a better way like a call back that can be written within the worker that will be called when job fails?

2

If you mean that you want to automatically retry it based on some criteria, then telemetry + retry is the exact mechanism you need. The telemetry events run within the job’s process and are provided in lieu of various hooks.

If this is something you want to do periodically, you can use a cron job that retries for you. Or, if it is something that is more manual based on some inspection, you can retry using the database queries (or the web dashboard).

1 Like
3

I guess my question is not clearly framed. My requirement is like below:

  • I have a table called job_a_tracker with fields like id, failed_job_id, status and the job called heavy_job.
  • Before i trigger the heavy_job I will create a record in job_a_tracker with status="started" & then trigger it.
  • Now when job completes successfully I can very well update the status in job_a_tracker by keeping it as a last statement in the worker.
  • But tricky part is when job gets failed. I need to save the failed_job_id and update status as Failed.

Right now I attached something like this in application.ex file:

:telemetry.attach(
      "oban-logger",
      [:oban, :job, :exception],
      &MyApp.JobTracker.handle_event/4,
      nil
    )

defmodule MyApp.JobTracker do
  require Logger
  def handle_event(
        [:oban, :job, :exception],
        %{duration: duration},
        %{
          attempt: attempt,
          max_attempts: max_attempts,
          worker: "MyApp.HeavyJob"
        } = meta,
        nil
      ) do
# update record with failed_job_id = meta.id & status = "Failed"
end
end

Is there a better way to handle this ?

1 Like
4

Nope. That is exactly how you’d handle that!

Now, I’m not sure why you’re double-tracking records in the DB, but that’s entirely your prerogative :slightly_smiling_face:

3 Likes
5

Oh I did this just to inform user that the job user submitted is success or failure. If failure option to retry the exact job. Is there a way to do this without double tracking?

6

Absolutely. The job is stored in the database within the oban_jobs table and has a state field on it. You can link to them or even set up foreign keys to track the job. The telemetry events will also let you immediately notify a user that the job succeeded or failed.

1 Like
7

Yeah that’s nice idea. If it is one to one. I mean one task needs one job. But in my case one task triggers multiple jobs. So In that case I need to have separate table with one to many(main_table_with_jobs_table). Every time I render the page I have to join 3 tables (main table, one to many table, oban table) to see all the information which might not be the fast thing to render :frowning: .

So I guess I can stick with previous solution itself by double tracking :slight_smile: .

8

I am accepting this as a solution. Since you made me clear that this is the way it needs to be done.

9

There’s probably a small gotcha in using oban_jobs table if the Oban.Plugins.Pruner is in use: that removes the old jobs from the table. I’m using the plugin to keep the jobs table clean and for that reason have another table where I track job state (especially discard/failure) for the same reason as @pavancse17.

1 Like