Hey folks! We’ve switched to using the new chain worker, but we’re ending up in a situation where it seems like the jobs are all ending up in a 'scheduled' state with a schedule date all way in the future.

What’s going on with this? How can we unstick it? We’re at like 600k+ jobs in scheduling right now. For the moment I’ve been doing:

update oban_jobs set scheduled_at=now(), state = 'available', meta = jsonb_set(meta, '{on_hold}', 'false', true) where id in (
select distinct on (args['shipment_id']) id from oban_jobs where queue = 'firehose' and state = 'scheduled' order by args['shipment_id'], args['event_id'] asc
)

to basically get the earliest event for each shipment and force enqueue the job. This seems pretty hacky though.

EDIT: This doesn’t actually work all that well to unstick stuff. Not sure what’s going on but the runtimes on the jobs are quite high, (~2 seconds to do a ~40ms push to cloud pubsub) and the number of running jobs at any moment seems low.

EDIT 2: Forgot to remove the global partitioning config from the queue. Problem solved!