Oban Stops Processing available Jobs

Im running into an issue with Oban where it will just stop processing jobs, even though they are marked as available in the database. I have an hourly recurring job (not through Cron) that will stop after the first time it is processed. I am able to reproduce this 100% of the time.

Edit: Dropping the time to 5 minutes instead of 1 hour causes the issue to not happen, at least quickly enough for me to notice.

This stops processing of all jobs in that queue (I only have a single type of job so I don’t know if all queues are stopped).

Its a standard Phoenix App. The only non-standard part of my setup is that I am using TimescaleDB as the database.

Code for the worker:

defmodule OldSchoolSnitch.Workers.HiScoreWorker do
  alias OldSchoolSnitch.Runescape
  require Logger
  @one_hour 60 * 60
  use Oban.Worker,
    queue: :hiscores,
    unique: [fields: [:args, :worker], states: [:available, :scheduled], period: @one_hour]

  @impl Oban.Worker
  def perform(%Oban.Job{
        args: %{"runescape_name" => runescape_name, "player_id" => player_id} = args,
        attempt: 1
      }) do
    args
    |> new(schedule_in: @one_hour)
    |> Oban.insert!()

    do_job(runescape_name, player_id)
  end

  def perform(%Oban.Job{
        args: %{"runescape_name" => runescape_name, "player_id" => player_id} = _args
      }) do
    do_job(runescape_name, player_id)
  end

  defp do_job(runescape_name, player_id) do
    Logger.debug("Running hi score job for #{runescape_name}")

    hi_scores = Runescape.HiScoresClient.get_hiscores(runescape_name)
    db_totals = Runescape.XPDrop.get_users_skill_totals(player_id)

    Enum.map(hi_scores, fn x ->
      skill = get_db_total_for_skill(db_totals, x.skill)

      case skill do
        nil ->
          %{
            player_id: player_id,
            skill: x.skill,
            amount: 0,
            current_total: x.experience
          }
          |> Runescape.XPDrop.create_xp_drop()

        _ ->
          if x.experience > skill.current_total do
            {:ok, drop} =
              %{
                player_id: player_id,
                skill: x.skill,
                amount: x.experience - skill.current_total,
                current_total: x.experience
              }
              |> Runescape.XPDrop.create_xp_drop()

            OldSchoolSnitchWeb.Endpoint.broadcast(drop.player_id, "step", drop)
          end
      end
    end)

    :ok
  end

  defp get_db_total_for_skill(db_totals, skill) do
    Enum.find(db_totals, fn x ->
      x.skill == skill
    end)
  end
end

I don’t think this would matter, but I am curious if your oban_jobs table is set up as a hypertable?

Negative. Just confirmed it is not a hypertable.

Do all jobs stop processing or just this one? Have you looked at the troubleshooting guide about this topic? Troubleshooting — Oban v2.13.6

All jobs in the queue stop processing. They become available and are not executed. I only have this one queue, so if needed I can add some others that do nothing but log a message.

I checked the guide, but I am not using PgBouncer and this is a single node setup. I have also confirmed that the Stager plugin is running (at least it shows up in LiveDashboard) at start. I can test the PG Notifier and the Repeater plugin, but that will be a while since it takes an hour of waiting to reproduce the issue.

Thanks!

1 Like

Minor update: Switching the notifier to using PG appeared to have resolved the problem. Im working on getting timescale set up on a different server to see if its a “my machine” issue or an actual issue with Timescale.

Weird! FWIW we have been using a timescaledb database for several years, and Oban since it was released, with no issue.

Thankfully, these “queues stop processing” issues will disappear with the next Oban release thanks to these changes: Automatically fall back to polling mode when PubSub isn't functional · Issue #769 · sorentwo/oban · GitHub