Oban - Reliable and Observable Job Processing

One suggestion - maybe instead of storing data as a JSONB field instead use binary field type and ETF to store arguments. It will provide you greater flexibility (for example distinction between atom and binary).

That was my initial inclination as well, but I ended up using JSONB instead of a few reasons:

1. It makes it much easier to enqueue jobs in other languages. The primary system I work on uses Elixir, Python and Ruby on the backend. It is essential that jobs can be enqueued from outside of Elixir/Erlang.
2. Searching and filtering is an important part of the UI and historic observation. By storing arguments as JSONB we can actually leverage indexes for full text search. A common situation we have is trying to determine if a job was ran for a particular customer or with particular arguments.

Thanks for the feedback!

This looks really great - good work! Looking forward to trying it out.

Wow. It looks great. I’ll take a looked here. Gratz

I’m glad you went for JSONB rather than ETF for serialisation.

Rihanna uses ETF and it has repeatedly made gathering data and altering jobs a difficult process that involves loading every row into application memory when I could have run a single SQL statement.

Is the manner in which Oban polls the database documented?

How is failure detection implemented? If you’re using Ecto it suggests that your not using transactions to detect worker death?

I am the weird guy who is amazed by job queues and background processing and I am kinda hyped about this library. I am sad I have no active use case for a job queue at the moment.

First, I’m very excited to see this! We have a similar home rolled solution but this has all kinds of wonderful features I’d love to have. The argument for using jsonb is sound, people should keep in mind that if there are specific values you want to keep as erlang terms you can always term |> :erlang.term_to_binary |> Base.encode64 manually on that term and put it in the job JSON, OR have those values in a different table that the job json points to.

PG_NOTIFY FYI

I’m noticing you’re using pg_notify I want to make sure you’re aware of an important detail in how pg_notify behaves, because I only learned about it recently and found it quite unexpected. For every invocation of pg_notify within a transaction, pg_notify dedups the notice against existing notices that are set to go out when the transaction commits. This check is O(n^2)!!!. This can cause serious issues if for example you encode the entire job payload in the notice, because 1000 jobs will create 1000 unique notices, and suddenly what was a simple transactional insert of 1000 records has taken several seconds.

Based on a quick glance at the code though it looks like you’re limiting pg_notify usage to just push out {"queue":queue, "state":job_state} json blobs which should be fine, since there are a limited number of queues and a limited number of job states.

Details here:

https://www.postgresql.org/message-id/flat/CAHg_5goMB8TJLBSB%3DsK9cX2CQhi%3DTtzGW1Ph47yev%2ByVgAqgyA%40mail.gmail.com#d3c67e942f11b5118a3081ba1792d979

looks great.. any pointers on pros/cons vs ecto_job | Hex and rihanna | Hex ?

Glad you can vouch for that choice as well!

Only slightly, in the config documentation for Oban.start_link/1. For most job dispatching it relies on triggers/notifications, but because of scheduled jobs it also polls every one second by default. The poll interval is effectively the scheduled job resolution.

That’s correct, it doesn’t use transactions at all. I routinely run jobs that can last for 15-30 minutes (video encoding, creating zips, etc) and holding a transaction for that long would eat up the connection pool.

There are two modes of failure that it handles:

1. Standard catch/rescue — The catch/rescue bit is built on top of telemetry. It uses a notification handler to enqueues a retry or discard dead jobs.
2. Shutdown (Orphans) — Every job acquires an advisory lock when it starts executing. The advistory lock is used to indicate which jobs are actively executing and which really dead. When a node shuts down it takes its connection pool with hit, which cleans up lingering advisory locks. Each node periodically scans for orphaned jobs and marks them as available again for retried execution.

It is easy to tell how effective the orphan rescue is because jobs are kept around after they are complete. Any job that has multiple attempts and no errors must have been orphaned!

If a job takes a lock when it starts how is the lock established for retrying a failed job (not a node death)? Wouldn’t you need to release the lock beforehand?