Long wait times when adding a big number of Oban jobs

Special Sections Oban
, ,
1

Hi! I’m dealing with long wait times (and requests getting dropped) due to a lock when we get a huge spike of job inserts. Is there any documentation on perf characteristics, min requirements, or perf tuning that I can look at? The purple is all Lock:object

If it matters, these are all chunk jobs.

image
image720×187 29.2 KB

I’m on PG 13.12, Oban 2.17, and Oban Pro 1.4.9

2

The top item is from pg_notify called by the application. There are a few things you can do to minimize or even eliminate that load:

  1. Switch from Oban.Notifiers.Postgres to Oban.Notifiers.PG if you happen to be clustered. It gets a lot of extra load off of the database.
  2. Disable insert triggers to prevent emitting a notification after each insert.
  3. If you can’t switch notifiers and don’t want to lose insert triggers, ensure you’re using bulk inserts with insert_all and avoid using unique where possible.
3

Thanks @sorentwo! Are insert triggers configurable by queues by any chance?

4

No, because they’re actually emitted from client code as jobs are inserted. However, you could easily replicate the trigger effect from client code if desired:

Oban.Notifier.notify(:insert, %{queue: "queue-name"})
2 Likes
5

We pushed the recommended changes to prod this morning and we haven’t seen any DB timeouts! Thanks again

1 Like