benwilson512
1.5 Chain worker backing up in 'scheduled' state
Hey folks! We’ve switched to using the new chain worker, but we’re ending up in a situation where it seems like the jobs are all ending up in a 'scheduled' state with a schedule date all way in the future.
What’s going on with this? How can we unstick it? We’re at like 600k+ jobs in scheduling right now. For the moment I’ve been doing:
update oban_jobs set scheduled_at=now(), state = 'available', meta = jsonb_set(meta, '{on_hold}', 'false', true) where id in (
select distinct on (args['shipment_id']) id from oban_jobs where queue = 'firehose' and state = 'scheduled' order by args['shipment_id'], args['event_id'] asc
)
to basically get the earliest event for each shipment and force enqueue the job. This seems pretty hacky though.
EDIT: This doesn’t actually work all that well to unstick stuff. Not sure what’s going on but the runtimes on the jobs are quite high, (~2 seconds to do a ~40ms push to cloud pubsub) and the number of running jobs at any moment seems low.
EDIT 2: Forgot to remove the global partitioning config from the queue. Problem solved!
Marked As Solved
sorentwo
This issue should be resolved in Pro v1.5.5 (seems likely based on outside feedback from @benwilson512). It will also be fixed in the upcoming Pro v1.6.0-rc.6 release.
Thanks for helping us get this sorted out!
Also Liked
benwilson512
As a quick followup for posterity: We did not end up rolling back. There’s a race condition in Oban with the chain worker where the query to find the next job to execute can race with a job being inserted at that moment. We have the unsticker code above running every 10 seconds or so which keeps things moving on our end. The Oban team is investigating solutions to the race condition.
benwilson512
Can confirm, everything has been running happily all day. Thanks for the quick turn around!
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









