Oban Pro v1.5.0 Released

It’s official, the release candidate phase is over.

Half a year and many moons have passed! Oban Pro v1.5 is generally available.

Read through the full release notes and history, or peruse the highlights below.

This release includes the new job decorator, unified migrations, a index-backed simple unique mode, changes for distributed PostgreSQL, improved batches, streamlined chains, worker aliases, hybrid job composition, and other performance improvements.

Job Decorator

The new Oban.Pro.Decorator module converts functions into Oban jobs with a teeny-tiny @job true annotation. Decorated functions, such as those in contexts or other non-worker modules, can be executed as fully fledged background jobs with retries, priority, scheduling, uniqueness, and all the other guarantees you have come to expect and love from Oban jobs.

defmodule Business do
  use Oban.Pro.Decorator

  @job max_attempts: 3, queue: :notifications
  def activate(account_id) when is_integer(account_id) do
    case Account.fetch(account_id) do
      {:ok, account} ->
        account
        |> notify_admin()
        |> notify_users()

      :error ->
        {:cancel, :not_found}
    end
  end
end

# Insert a Business.activate/1 job
Business.insert_activate(123)

The @job decorator also supports most standard Job options, validated at compile time. As expected, the options can be overridden at runtime through an additional generated clause. Along with generated insert_ functions, there’s also a new_ variant that be used to build up job changesets for bulk insert, and a relay_ variant that operates like a distributed async/await.

Finally, the generated functions also respect patterns and guards, so you can write assertive clauses that defend against bad inputs or break logic into multiple clauses.

Unified Migrations

Oban has had centralized, versioned migrations from the beginning. When there’s a new release with database changes, you run the migrations and it figures out what to change on its own. Pro behaved differently for reasons that made sense when there was a single producers table, but don’t track with multiple tables and custom indexes.

Now Pro has unified migrations to keep all the necessary tables and indexes updated and fresh, and you’ll be warned at runtime if the migrations aren’t current.

See the Oban.Pro.Migration module for more details, or check the v1.5 Upgrade Guide for instructions on putting it to use.

Enhanced Unique

Oban’s standard unique options are robust, but they require multiple queries and centralized locks to function. Now Pro supports an enhanced unique mode designed for speed, correctness, and scalability.

The enhanced mode boosts insert performance 1.5x-3.5x, reduces database load with fewer queries, improves memory usage, and remains correct across multiple processes/nodes.

Here’s a comparison between inserting various batches with the legacy and enhanced modes:

See more in the Enhanced Unique section.

Distributed PostgreSQL

There were a handful of PostgreSQL features used in Oban and Pro that prevented it from running in distributed PostgreSQL clients such as CockroachDB or Yugabyte.

A few table creation options prevented even running the migrations due to unsupported database features. Then there were advisory locks, which are part of how Oban normally handles unique jobs, and how Pro coordinates queues globally.

We’ve worked around all of these limitations and it’s now possible to run Oban and Pro on CockroachDB and Yugabyte with all of the same functionality as regular PostgreSQL (unique jobs, global, rate limits, queue partitioning).

Improved Batches

One of the Pro’s original three features, batches link the execution of many jobs as a group and run optional callback jobs after jobs are processed.

Composing batches used to rely on a dedicated worker, one that couldn’t be composed with other worker types. Now, there’s a stand alone Oban.Pro.Batch module that’s used to dynamically build, append, and manipulate batches from any type of job, and with much more functionality.

Batches gain support for streams (creating and appending with them), clearer callbacks, and allow setting any Oban.Job option on callback jobs.

alias Oban.Pro.Batch

mail_jobs = Enum.map(mail_args, &MyApp.MailWorker.new/1)
push_jobs = Enum.map(push_args, &MyApp.PushWorker.new/1)

[callback_opts: [priority: 9], callback_worker: CallbackWorker]
|> Batch.new()
|> Batch.add(mail_jobs)
|> Batch.add(push_jobs)

Learn more in the new Batch module docs.

Streamlined Chains

Chains now operate like workflows, where jobs are scheduled until they’re ready to run and then descheduled after the previous link in the chain completes. Preemptive chaining doesn’t clog queues with waiting jobs, and it chews through a backlog without any polling.

Chains are also a standard Oban.Pro.Worker option now. There’s no need to define a chain specific worker, in fact, doing so is deprecated. Just add the chain option and you’re guaranteed a FIFO chain of jobs:

-  use Oban.Pro.Workers.Chain, by: :worker
+  use Oban.Pro.Worker, chain: [by: :worker]

See more in the Chained Jobs section.

Improved Workflows

Workflows began the transition from a dedicated worker to a stand-alone module several versions ago. Now that transition is complete, and workflows can be composed from any type of job.

All workflow management functions have moved to a centralized Oban.Pro.Workflow module. An expanded set of functions, including the ability to cancel an entire workflow, conveniently work with either a workflow job or id, so it’s possible to maneuver workflows from anywhere.

Perhaps the most exciting addition, because it’s visual and we like shiny things, is the addition of mermaid output for visualization. Mermaid has become the graphing standard, and it’s an excellent way to visualize workflows in tools like LiveBook.

alias Oban.Pro.Workflow

workflow =
  Workflow.new()
  |> Workflow.add(:a, EchoWorker.new(%{id: 1}))
  |> Workflow.add(:b, EchoWorker.new(%{id: 2}), deps: [:a])
  |> Workflow.add(:c, EchoWorker.new(%{id: 3}), deps: [:b])
  |> Workflow.add(:d, EchoWorker.new(%{id: 4}), deps: [:c])

Oban.insert_all(workflow)

Workflow.cancel_jobs(workflow.id)

Learn more in the new Workflow module docs.

Worker Aliases

Worker aliases solve a perennial production issue—how to rename workers without breaking existing jobs. Aliasing allows jobs enqueued with the original worker name to continue executing without exceptions using the new worker code.

-defmodule MyApp.UserPurge do
+defmodule MyApp.DataPurge do
-  use Oban.Pro.Worker
+  use Oban.Pro.Worker, aliases: [MyApp.UserPurge]

See more in the Worker Aliases section.