Oban cancel callbacks - is there a way to provide code to execute if an Oban worker is cancelled?

Special Sections Oban
,
1

Is there a way to provide code to execute if an Oban worker is cancelled? It’s not listed as one of the callbacks provided by the relevant behaviour, but there might be another way I’m missing?

1 Like
2

:wave: @benswift

I hope someone would post a better approach, but I’ve seen Oban.Telemetry — Oban v2.18.3 being used to run callbacks like this.

defmodule MyApp.ObanTelemetry do
  require Logger

  def handle_event([:oban, :job, :stop], _measures, meta, _config) do
    if meta.state == :cancelled do
      your_cancel_callback(meta.job)
    end
  catch
    kind, reason ->
      message = Exception.format(kind, reason, __STACKTRACE__)
      Logger.error(message)
  end
end

:telemetry.attach("oban-job-stop", [:oban, :job, :stop], &MyApp.ObanTelemetry.handle_event/4, nil)

Just note that if the callback raises, the telemetry handler would be detached. Hence the catch clause.

2 Likes
3

Yep, thanks for the tip.

To be honest because all the call-sites for cancelling the jobs are in my code, I think I might just make a helper function which does the Oban.cancel call and my own code for cleaning up. Using telemetry is an interesting (if, as you say, a bit hacky) option, though.

4

There are three situations where a job may be cancelled:

  1. By returning a {:cancel, reason} tuple from your perform/1 callback
  2. During execution, from Oban.cancel_job called programmatically or through the web dashboard
  3. Not during execution, also from Oban.cancel_job

The first and second variants will emit an [:oban, :job, :stop] as mentioned by @ruslandoga. However, the third won’t emit a :stop event because the job isn’t running at that point.

If your goal is to respond to the job being cancelled during execution, then the telemetry approach or Pro’s worker hooks are a good approach.

This leads me to believe that you’re mostly concerned with handling situations two and three from the list above, where cancel_job is used. In that case, there are engine telemetry events that include information about the cancelled job (id, queue, state).

That’s an indirect way to handle cancel cleanup though, and if you’re in control of the cancellation I’d favor wrapping the cancel call and the cleanup in a transaction.

2 Likes
5

Thanks @sorentwo , yep, I’m in control of the cancellation - and I agree that just wrapping in a transaction is better than (ab)using the telemetry for that purpose. Thanks for your advice (and thanks for Oban which is awesome).

1 Like
6

I have similar question. Is there way to see in development console errors when Job fail? I mean during app development on dev computer I don’t see any errors when job fail or even crash. Is it required to have custom telemetry handler module to show it?

using something like this

config :my_app, Oban, log: :error (or other values)

is super noisy and doesn’t fully tell that job failed.

7

Absolutely! Enable the default logger with Telemetry.attach_default_logger/1. That will log information about the job, including the exception and other failure details. It’s very handy, and mentioned in the instrumentation section as well as the preparing for production guide.

The log attribute is poorly named and only applies to Ecto queries. It should be named something like ecto_log or repo_log (it’s tracked changing in 3.0).

8

Thanks, problem is that we use bug tracking tool/service which handle these errors in production. So practically we have to create something like this which will be applied only in local development?

9

Are you opposed to logging and error reporting? There is other valuable information in the logs beyond exceptions, e.g. job timing, pubsub connectivity, job staging issues.

If you really don’t want to enable logging in prod, then you can disable it conditionally or set the log level to something that won’t show up:

Oban.Telemetry.attach_default_logger(level: Application.get_env(:my_app, :oban_log_level, :debug))
2 Likes