Oban Job does not retry, returns: {:error, %{}

iex(15)> DateTime.utc_now()
~U[2024-08-06 15:38:51.337192Z]
iex(16)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, [%{id: 350, state: "scheduled", queue: "delivery_destination"}]}
iex(17)>

job is scheduled for: 2024-08-06 15:38:48.665877

The 1 stage_job we are seeing here is a different job. We send 2 emails and each one goes to a different location. 1 of those email destinations, I am using schedule_in or failing on purpose, and that is the one that never runs or retries.

After the first email job is completed, the stage_job command returns an empty list:

iex(15)> DateTime.utc_now()
~U[2024-08-06 15:38:51.337192Z]
iex(16)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, [%{id: 350, state: "scheduled", queue: "delivery_destination"}]}
iex(17)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, []}
iex(18)> 

What’s the attempt and max_attempts for that job?

attempt is 0 and max is 3

Nothing appears to be wrong and I don’t see any reason why a scheduled job wouldn’t be ran. Staging is clearly working, as demonstrated by the stage call.

Will you run through this set of queries and see if it matches what you expect?

import Ecto.Query

# Check how many are scheduled or retryable
Oban.Job |> where([j], j.state in ~w(scheduled retryable)) |> select(count()) |> Repo.one()

# Add the queue check
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> select(count())
|> Repo.one()

# Find the max time
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> select([j], max(j.scheduled_at)))
|> Repo.one()

# Add the time check
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> where([j] j.scheduled_at <= ^DateTime.utc_now())
|> select(count())
|> Repo.one()

iex(35)> import Ecto.Query
Ecto.Query
iex(36)> alias Engine.Repo
Engine.Repo
iex(37)> Oban.Job |> where([j], j.state in ~w(scheduled retryable)) |> select(count()) |> Repo.one()
34
iex(38)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> select(count()) |> Repo.one()
34
iex(39)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> select([j], max(j.scheduled_at)) |> Repo.one()
~U[2024-08-07 17:04:37.664335Z]
iex(40)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> where([j], j.scheduled_at <= ^DateTime.utc_now()) |> select(count()) |> Repo.one()
34
iex(41)>

So there are 34 scheduled jobs that are ready to go? If you run stage_jobs as you did earlier, does it stage any of them?

That’s approximately the same query that stage_jobs uses, so I wouldn’t expect any difference.

looks like it

iex(62)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok,
 [
   %{id: 1, state: "scheduled", queue: "recalc"},
   %{id: 2, state: "scheduled", queue: "recalc"},
   %{id: 3, state: "scheduled", queue: "recalc"},
   %{id: 4, state: "scheduled", queue: "recalc"},
   %{id: 5, state: "scheduled", queue: "recalc"},
   %{id: 6, state: "scheduled", queue: "recalc"},
   %{id: 7, state: "scheduled", queue: "recalc"},
   %{id: 13, state: "scheduled", queue: "recalc"},
   %{id: 14, state: "scheduled", queue: "recalc"},
   %{id: 16, state: "scheduled", queue: "recalc"},
   %{id: 17, state: "scheduled", queue: "recalc"},
   %{id: 18, state: "scheduled", queue: "recalc"},
   %{id: 19, state: "scheduled", queue: "recalc"},
   %{id: 20, state: "scheduled", queue: "recalc"},
   %{id: 25, state: "scheduled", queue: "recalc"},
   %{id: 26, state: "scheduled", queue: "recalc"},
   %{id: 27, state: "scheduled", queue: "recalc"},
   %{id: 28, state: "scheduled", queue: "recalc"},
   %{id: 29, state: "scheduled", queue: "recalc"},
   %{id: 30, state: "scheduled", queue: "recalc"},
   %{id: 31, state: "scheduled", queue: "recalc"},
   %{id: 32, state: "scheduled", queue: "recalc"},
   %{id: 33, state: "scheduled", queue: "recalc"},
   %{id: 34, state: "scheduled", queue: "recalc"},
   %{id: 35, state: "scheduled", queue: "recalc"},
   %{id: 36, state: "scheduled", queue: "recalc"},
   %{id: 37, state: "scheduled", queue: "recalc"},
   %{id: 38, state: "scheduled", queue: "recalc"},
   %{id: 39, state: "scheduled", queue: "recalc"},
   %{id: 40, state: "scheduled", queue: "recalc"},
   %{id: 42, state: "scheduled", queue: "process_status"},
   %{id: 43, state: "retryable", queue: "delivery_destination"},
   %{id: 47, state: "scheduled", queue: "process_status"},
   %{id: 52, state: "scheduled", queue: "process_status"}
 ]}
iex(63)>

@houayang did you find a resolution to this issue? I’ve found myself dealing with a similar issue. Following this thread and trying things as prescribed gets me similar results.

If I use Oban.Engine.stage_jobs(Oban.config(), Oban.Job, limit: 5000) directly the jobs are staged but otherwise they hang out.

Which version of Oban and what notifier are you using? Newer versions will emit a warning in the logger when nodes aren’t connected, and should fall back to a local polling mechanism.

Howdy @sorentwo Thanks for the super fast response!

Oban 2.18.3 and Oban.Notifiers.Postgres.

This is local development. It’s a very basic Oban config