It’s official, the release candidate phase is over.
Half a year and many moons have passed! Oban Pro v1.5 is generally available.
Read through the full release notes and history, or peruse the highlights below.
This release includes the new job decorator, unified migrations, a index-backed simple unique mode, changes for distributed PostgreSQL, improved batches, streamlined chains, worker aliases, hybrid job composition, and other performance improvements.
Job Decorator
The new Oban.Pro.Decorator module converts functions into Oban jobs with a teeny-tiny @job true
annotation. Decorated functions, such as those in contexts or other non-worker modules, can be executed as fully fledged background jobs with retries, priority, scheduling, uniqueness, and all the other guarantees you have come to expect and love from Oban jobs.
defmodule Business do
use Oban.Pro.Decorator
@job max_attempts: 3, queue: :notifications
def activate(account_id) when is_integer(account_id) do
case Account.fetch(account_id) do
{:ok, account} ->
account
|> notify_admin()
|> notify_users()
:error ->
{:cancel, :not_found}
end
end
end
# Insert a Business.activate/1 job
Business.insert_activate(123)
The @job
decorator also supports most standard Job options, validated at compile time. As expected, the options can be overridden at runtime through an additional generated clause. Along with generated insert_ functions, there’s also a new_ variant that be used to build up job changesets for bulk insert, and a relay_ variant that operates like a distributed async/await.
Finally, the generated functions also respect patterns and guards, so you can write assertive clauses that defend against bad inputs or break logic into multiple clauses.
Unified Migrations
Oban has had centralized, versioned migrations from the beginning. When there’s a new release with database changes, you run the migrations and it figures out what to change on its own. Pro behaved differently for reasons that made sense when there was a single producers
table, but don’t track with multiple tables and custom indexes.
Now Pro has unified migrations to keep all the necessary tables and indexes updated and fresh, and you’ll be warned at runtime if the migrations aren’t current.
See the Oban.Pro.Migration
module for more details, or check the v1.5 Upgrade Guide for instructions on putting it to use.
Enhanced Unique
Oban’s standard unique options are robust, but they require multiple queries and centralized locks to function. Now Pro supports an enhanced unique mode designed for speed, correctness, and scalability.
The enhanced mode boosts insert performance 1.5x-3.5x, reduces database load with fewer queries, improves memory usage, and remains correct across multiple processes/nodes.
Here’s a comparison between inserting various batches with the legacy and enhanced modes:
jobs | legacy | enhanced | boost |
---|---|---|---|
100 | 45.08 | 33.93 | 1.36 |
1000 | 140.64 | 81.452 | 1.72 |
10000 | 3149.71 | 979.47 | 3.22 |
20000 | oom error | 1741.67 |
See more in the Enhanced Unique section.
Distributed PostgreSQL
There were a handful of PostgreSQL features used in Oban and Pro that prevented it from running in distributed PostgreSQL clients such as CockroachDB or Yugabyte.
A few table creation options prevented even running the migrations due to unsupported database features. Then there were advisory locks, which are part of how Oban normally handles unique jobs, and how Pro coordinates queues globally.
We’ve worked around all of these limitations and it’s now possible to run Oban and Pro on CockroachDB and Yugabyte with all of the same functionality as regular PostgreSQL (unique jobs, global, rate limits, queue partitioning).
Improved Batches
One of the Pro’s original three features, batches link the execution of many jobs as a group and run optional callback jobs after jobs are processed.
Composing batches used to rely on a dedicated worker, one that couldn’t be composed with other worker types. Now, there’s a stand alone Oban.Pro.Batch
module that’s used to dynamically build, append, and manipulate batches from any type of job, and with much more functionality.
Batches gain support for streams (creating and appending with them), clearer callbacks, and allow setting any Oban.Job
option on callback jobs.
alias Oban.Pro.Batch
mail_jobs = Enum.map(mail_args, &MyApp.MailWorker.new/1)
push_jobs = Enum.map(push_args, &MyApp.PushWorker.new/1)
[callback_opts: [priority: 9], callback_worker: CallbackWorker]
|> Batch.new()
|> Batch.add(mail_jobs)
|> Batch.add(push_jobs)
Learn more in the new Batch module docs.
Streamlined Chains
Chains now operate like workflows, where jobs are scheduled until they’re ready to run and then descheduled after the previous link in the chain completes. Preemptive chaining doesn’t clog queues with waiting jobs, and it chews through a backlog without any polling.
Chains are also a standard Oban.Pro.Worker
option now. There’s no need to define a chain specific worker, in fact, doing so is deprecated. Just add the chain
option and you’re guaranteed a FIFO chain of jobs:
- use Oban.Pro.Workers.Chain, by: :worker
+ use Oban.Pro.Worker, chain: [by: :worker]
See more in the Chained Jobs section.
Improved Workflows
Workflows began the transition from a dedicated worker to a stand-alone module several versions ago. Now that transition is complete, and workflows can be composed from any type of job.
All workflow management functions have moved to a centralized Oban.Pro.Workflow
module. An expanded set of functions, including the ability to cancel an entire workflow, conveniently work with either a workflow job or id, so it’s possible to maneuver workflows from anywhere.
Perhaps the most exciting addition, because it’s visual and we like shiny things, is the addition of mermaid output for visualization. Mermaid has become the graphing standard, and it’s an excellent way to visualize workflows in tools like LiveBook.
alias Oban.Pro.Workflow
workflow =
Workflow.new()
|> Workflow.add(:a, EchoWorker.new(%{id: 1}))
|> Workflow.add(:b, EchoWorker.new(%{id: 2}), deps: [:a])
|> Workflow.add(:c, EchoWorker.new(%{id: 3}), deps: [:b])
|> Workflow.add(:d, EchoWorker.new(%{id: 4}), deps: [:c])
Oban.insert_all(workflow)
Workflow.cancel_jobs(workflow.id)
Learn more in the new Workflow module docs.
Worker Aliases
Worker aliases solve a perennial production issue—how to rename workers without breaking existing jobs. Aliasing allows jobs enqueued with the original worker name to continue executing without exceptions using the new worker code.
-defmodule MyApp.UserPurge do
+defmodule MyApp.DataPurge do
- use Oban.Pro.Worker
+ use Oban.Pro.Worker, aliases: [MyApp.UserPurge]
See more in the Worker Aliases section.