Some of my Oban (v2.19.1) jobs are somewhat rarely not being removed from its queue with the job remaining in the list of running jobs according to Oban.check_all_queues, however this is only impacting one of the two queues at the moment. In this case, it’s a job that’s in the completed state and was completed hours before I checked the queue, but I have also seen it occur on jobs that are discarded (presumably by Lifeline). As expected, once the queue’s list of running jobs reaches the limit, jobs for that queue stop being processed. What could I do to debug this?
iex> Oban.check_all_queues
[
%{
name: Oban,
node: "tornium@ubuntu-16gb-fsn1-1",
running: [323009],
queue: "notifications",
started_at: ~U[2025-02-28 03:06:04.182443Z],
limit: 20,
uuid: "ee9f0d07-ad88-4569-a861-95ad0c2a0a16",
updated_at: ~U[2025-02-28 23:27:24.183233Z],
paused: false,
refresh_interval: 30000,
shutdown_started_at: nil
},
%{
name: Oban,
node: "tornium@ubuntu-16gb-fsn1-1",
running: [],
queue: "scheduler",
started_at: ~U[2025-02-28 03:06:04.182585Z],
limit: 10,
uuid: "5afa72ae-ff08-445e-806c-6c8e5ee181d5",
updated_at: ~U[2025-02-28 23:27:26.825306Z],
paused: false,
refresh_interval: 30000,
shutdown_started_at: nil
}
]
iex> Oban.Job |> Tornium.Repo.get(323009)
%Oban.Job{
__meta__: #Ecto.Schema.Metadata<:loaded, "oban_jobs">,
id: 323009,
state: "completed",
queue: "notifications",
worker: "Tornium.Workers.Notification",
args: %{
"notifications" => ["54a26d6f-764b-4f63-be8d-506770a8c251",
"ddb3e24e-16e9-48da-9836-c98066753920"],
"resource" => "faction",
"resource_id" => 51803
},
meta: %{},
tags: ["notification"],
errors: [],
attempt: 2,
attempted_by: ["tornium@ubuntu-16gb-fsn1-1",
"ee9f0d07-ad88-4569-a861-95ad0c2a0a16"],
max_attempts: 2,
priority: 5,
attempted_at: ~U[2025-02-28 17:44:11.809479Z],
cancelled_at: nil,
completed_at: ~U[2025-02-28 17:44:25.087712Z],
discarded_at: nil,
inserted_at: ~U[2025-02-28 17:43:00.949011Z],
scheduled_at: ~U[2025-02-28 17:43:00.949011Z],
conf: nil,
conflict?: false,
replace: nil,
unique: nil,
unsaved_error: nil
}
iex> Oban.config
%Oban.Config{
dispatch_cooldown: 5,
engine: Oban.Engines.Basic,
get_dynamic_repo: nil,
insert_trigger: true,
log: false,
name: Oban,
node: "tornium@ubuntu-16gb-fsn1-1",
notifier: {Oban.Notifiers.Postgres, []},
peer: {Oban.Peers.Database, []},
plugins: [
{Oban.Plugins.Lifeline, [rescue_after: 60000, interval: 30000]},
{Oban.Plugins.Pruner, [max_age: 86400]},
{Oban.Plugins.Cron,
[crontab: [{"* * * * *", Tornium.Workers.NotificationScheduler}]]}
],
prefix: "public",
queues: [notifications: [limit: 20], scheduler: [limit: 10]],
repo: Tornium.Repo,
shutdown_grace_period: 30000,
stage_interval: 1000,
testing: :disabled
}
