houayang

houayang OP 

iex(15)> DateTime.utc_now()
~U[2024-08-06 15:38:51.337192Z]
iex(16)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, [%{id: 350, state: "scheduled", queue: "delivery_destination"}]}
iex(17)>

job is scheduled for: 2024-08-06 15:38:48.665877

The 1 stage_job we are seeing here is a different job. We send 2 emails and each one goes to a different location. 1 of those email destinations, I am using schedule_in or failing on purpose, and that is the one that never runs or retries.

After the first email job is completed, the stage_job command returns an empty list:

iex(15)> DateTime.utc_now()
~U[2024-08-06 15:38:51.337192Z]
iex(16)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, [%{id: 350, state: "scheduled", queue: "delivery_destination"}]}
iex(17)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok, []}
iex(18)>
0
Post #11
sorentwo

sorentwo

Oban Core Team

What’s the attempt and max_attempts for that job?

0
Post #12
houayang

houayang OP

attempt is 0 and max is 3

0
Post #13
sorentwo

sorentwo

Oban Core Team

Nothing appears to be wrong and I don’t see any reason why a scheduled job wouldn’t be ran. Staging is clearly working, as demonstrated by the stage call.

Will you run through this set of queries and see if it matches what you expect?

import Ecto.Query

# Check how many are scheduled or retryable
Oban.Job |> where([j], j.state in ~w(scheduled retryable)) |> select(count()) |> Repo.one()

# Add the queue check
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> select(count())
|> Repo.one()

# Find the max time
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> select([j], max(j.scheduled_at)))
|> Repo.one()

# Add the time check
Oban.Job
|> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue))
|> where([j] j.scheduled_at <= ^DateTime.utc_now())
|> select(count())
|> Repo.one()
1
Post #14
houayang

houayang OP 

iex(35)> import Ecto.Query
Ecto.Query
iex(36)> alias Engine.Repo
Engine.Repo
iex(37)> Oban.Job |> where([j], j.state in ~w(scheduled retryable)) |> select(count()) |> Repo.one()
34
iex(38)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> select(count()) |> Repo.one()
34
iex(39)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> select([j], max(j.scheduled_at)) |> Repo.one()
~U[2024-08-07 17:04:37.664335Z]
iex(40)> Oban.Job |> where([j], j.state in ~w(scheduled retryable) and not is_nil(j.queue)) |> where([j], j.scheduled_at <= ^DateTime.utc_now()) |> select(count()) |> Repo.one()
34
iex(41)>
0
Post #15
sorentwo

sorentwo

Oban Core Team

So there are 34 scheduled jobs that are ready to go? If you run stage_jobs as you did earlier, does it stage any of them?

That’s approximately the same query that stage_jobs uses, so I wouldn’t expect any difference.

1
Post #16
houayang

houayang OP

looks like it

iex(62)> Oban.Engine.stage_jobs(Oban.config(Manifold.Oban), Oban.Job, limit: 5000)
{:ok,
 [
   %{id: 1, state: "scheduled", queue: "recalc"},
   %{id: 2, state: "scheduled", queue: "recalc"},
   %{id: 3, state: "scheduled", queue: "recalc"},
   %{id: 4, state: "scheduled", queue: "recalc"},
   %{id: 5, state: "scheduled", queue: "recalc"},
   %{id: 6, state: "scheduled", queue: "recalc"},
   %{id: 7, state: "scheduled", queue: "recalc"},
   %{id: 13, state: "scheduled", queue: "recalc"},
   %{id: 14, state: "scheduled", queue: "recalc"},
   %{id: 16, state: "scheduled", queue: "recalc"},
   %{id: 17, state: "scheduled", queue: "recalc"},
   %{id: 18, state: "scheduled", queue: "recalc"},
   %{id: 19, state: "scheduled", queue: "recalc"},
   %{id: 20, state: "scheduled", queue: "recalc"},
   %{id: 25, state: "scheduled", queue: "recalc"},
   %{id: 26, state: "scheduled", queue: "recalc"},
   %{id: 27, state: "scheduled", queue: "recalc"},
   %{id: 28, state: "scheduled", queue: "recalc"},
   %{id: 29, state: "scheduled", queue: "recalc"},
   %{id: 30, state: "scheduled", queue: "recalc"},
   %{id: 31, state: "scheduled", queue: "recalc"},
   %{id: 32, state: "scheduled", queue: "recalc"},
   %{id: 33, state: "scheduled", queue: "recalc"},
   %{id: 34, state: "scheduled", queue: "recalc"},
   %{id: 35, state: "scheduled", queue: "recalc"},
   %{id: 36, state: "scheduled", queue: "recalc"},
   %{id: 37, state: "scheduled", queue: "recalc"},
   %{id: 38, state: "scheduled", queue: "recalc"},
   %{id: 39, state: "scheduled", queue: "recalc"},
   %{id: 40, state: "scheduled", queue: "recalc"},
   %{id: 42, state: "scheduled", queue: "process_status"},
   %{id: 43, state: "retryable", queue: "delivery_destination"},
   %{id: 47, state: "scheduled", queue: "process_status"},
   %{id: 52, state: "scheduled", queue: "process_status"}
 ]}
iex(63)>
0
Post #17
doomspork

doomspork

@houayang did you find a resolution to this issue? I’ve found myself dealing with a similar issue. Following this thread and trying things as prescribed gets me similar results.

If I use Oban.Engine.stage_jobs(Oban.config(), Oban.Job, limit: 5000) directly the jobs are staged but otherwise they hang out.

0
Post #18
sorentwo

sorentwo

Oban Core Team

Which version of Oban and what notifier are you using? Newer versions will emit a warning in the logger when nodes aren’t connected, and should fall back to a local polling mechanism.

0
Post #19
doomspork

doomspork

Howdy @sorentwo :wave: Thanks for the super fast response! :dash:

Oban 2.18.3 and Oban.Notifiers.Postgres.

This is local development. It’s a very basic Oban config

0
Post #20