There are several possible reasons why jobs won’t run, but it seems specifically due to those timeout errors. Some immediate changes you can make to try and get the system unstuck:

1. Rebuild the indexes (reindex table oban_jobs), force analyze (vacuum analyze oban_jobs)

2. Increase the size of your RDS instance (it may be fine for normal usage, but not for a backlog of 20m jobs)

3. Manually move most of those jobs to a scheduled state to space them out. Something like this could work to reduce the available count down to 100k jobs:

   update oban_jobs set state = 'scheduled', scheduled_at = now() + '1 hour'::interval
   where id < (select min(id) from oban_jobs where state = 'available') - 100000
   

Beyond that, here are some other suggestions once the system is moving again:

1. Apply some of the options from the official scaling guide.
2. Upgrade to Oban v2.19 and Pro v1.5+, primarily for the new check_available/1 implementation that optimizes the query to check for available jobs (benchmark info in the linked changelog)
3. Upgrade to Oban Web v2.10+ (v2.11 if possible), because it uses Oban.Met for much less database impact (especially count estimates)