The peer, notifier, and stager events are all helpful for diagnosing whatever happened in your use-case.

This seems like an issue with postgres pubsub. A disconnect error in versions prior to v0.20 would leave the Oban.Notifier in an undetectable disconnected state. The node would keep emitting notifications about available jobs, but they wouldn’t broadcast.

There are several ways you could monitor this:

1. Use checkins in a tracker like sentry or honeybadger. Send an from a cron job every minute and get alerted if they stop running.
2. Use cron monitors in sentry, which have native support for oban in recent versions.
3. Poll jobs and send it to an external tool like grafana, datadog, etc. and set up alerts on those. If you’re using the PG notifier, then Oban.Met.latest/2 is a good option. Otherwise, query the actual database every minute or so.

Without a running instance it’s difficult to say what the problem may have been. The fact that restarting got things running again points to an ephemeral issue, rather than something with the jobs table itself.

That would be useful information because it would indicate whether the nodes were able to communicate with eachother, and whether the stager “thought” it was notifying queues about available jobs. Staging events are included in the default logger (and is recommended as part of the Ready for Production guide).