OSS Oban Web & Oban v2.19

The title doesn’t relay the depth of this announcement, but it is short, balanced, and looked good to us. From open sourcing multiple packages, to releases with features like MySQL support, to plan changes—it’s immense!

Open Sourcing Oban Web

:postal_horn: Oban Web is now open source and free (as in champagne :clinking_glasses:)! From now on, Oban Web will be published to Hex and available for use in all your Oban powered applications.

Why? Here’s some history.

Our original Oban announcement on the Elixir Forum included a screenshot of Oban Web. That was nearly six years ago now, and oy have we come a long way. Originally launched as a private beta, Web was our first foray into building a viable business model for Oban. Years ago we realized our customers needed a more complex set of tools. Thus, we released Oban Pro. Pro is where the serious business happens now, and we want more people to have Web available from the start.

If you’re a current customer, or wonder what this means for our existing subscriptions, bounce to the Simplified Plans section.

Let’s cover Oban v2.19 first, then get back to more Web news.

Oban v2.19

The latest Oban release includes a new database engine, an installer for improved UX, better logging, and more. Here are some of the highlights.

:dolphin: MySQL Support

Oban officially supports MySQL with the new Dolphin engine. Oban works with modern (read “with full JSON support”) MySQL versions from 8.4 on, and has been tested on the highly scalable Plantescale database.

Running on MySQL is as simple as specifying the Dolphin engine in your configuration:

config :my_app, Oban,
  engine: Oban.Engines.Dolphin,
  queues: [default: 10],
  repo: MyApp.Repo

With this addition, Oban can run in an estimated 10% more Elixir applications!

:alembic: Automated Installer

Installing Oban into a new application is simplified with a new igniter powered mix task. The new oban.install task handles installing and configuring a standard Oban installation, and it will deduce the correct engine and notifier automatically based on the database adapter.

mix igniter.install oban

This oban.install task is currently the recommended way to install Oban. As a bonus, the task composes together with other igniter installers, making it possible to install phoenix and oban with a single igniter.new command:

mix archive.install hex igniter_new

mix igniter.new oban_demo \
  --with phx.new \
  --with-args="--database postgres" \
  --install oban

:notebook_with_decorative_cover: Logging Enhancements

Logging in a busy system may be noisy due to job events, but there are other events that are particularly useful for diagnosing issues. A new events option for attach_default_logger/1 allows selective event logging, so it’s possible to receive important notices such as notifier connectivity issues, without logging all job activity:

Oban.Telemetry.attach_default_logger(events: ~w(notifier peer stager)a)

Along with filtering, there are new events to make diagnosing operational problems easier.

A peer:election event logs leadership changes to indicate when nodes gain or lose leadership. Leadership issues are rare, but insidious, and make diagnosing production problems especially tricky.

[
  message: "peer became leader",
  source: "oban",
  event: "peer:election",
  node: "worker.1",
  leader: true,
  was_leader: false
]

Activity for all official plugins is now logged via plugin:stop and plugin:exception events. That includes runtime information to help monitor activity and diagnose issues. For example, every time Cron runs successfully it will output details about the execution time and all of the inserted job ids:

[
  source: "oban",
  duration: 103,
  event: "plugin:stop",
  plugin: "Oban.Plugins.Cron",
  jobs: [1, 2, 3]
]

There are other notable changes such as the new Oban.check_all_queues/1 function, starting queues in parallel on init, and official JSON integration. Read more in the full Changelog.

Oban Web v2.11

Beyond preparations for open sourcing, Web v2.11 includes substantial work toward component cleanup and unification. Here are some of our favorite new features:

:dolphin::feather: MySQL and SQLite

All official engines are fully supported. Listing, filtering, ordering, and searching through jobs works for Postgres, MySQL, and SQLite. That includes the particularly gnarly issue of dynamically generating and manipulating JSON for filter auto-suggestions! Nested args queries, such as args.address.city:Edinburgh work equally well with each engine.

:control_knobs: Instance Select

The dashboard will now support switching between Oban instances.

A new instance select menu in the header allows switching between running Oban instances at runtime. There’s no need to mount additional dashboards in your application’s router to handle multiple instances. The most recently selected instance persists between mounts.

- oban_dashboard "/oban_a", oban_name: Oban.A
- oban_dashboard "/oban_b", oban_name: Oban.B
- oban_dashboard "/oban_c", oban_name: Oban.C
+ oban_dashboard "/oban"

This also eliminates the need for additional router configuration. The dashboard will select the first running Oban instance it finds (with a preference for the default Oban).

- oban_dashboard "/oban", oban_name: MyOban
+ oban_dashboard "/oban"

:yin_yang: Rebuilt Queues Page

The queue and jobs tables are fully rebuilt with shared, reusable components and matching functionality.

This enables far more powerful interaction with queues:

  • Uniform Navigation - click on any part of the queue row to navigate to details.

  • Sidebar - a new queue sidebar shows status counts and enables filtering by statuses such as paused or terminating.

  • Filtering - queues are auto-complete filterable just like jobs, making it possible to find queues running on a particular node or narrow down by status.

  • Shared Sorting - queue sorting now behaves identically to jobs, through a shared dropdown.

  • Condensed Rows - simplify the queue page by removing nested row components. Extra queue details are in the sub-queue page.

:candle: Operate on Full Selection

Apply bulk actions to all selected jobs, not just those visible on the current page.

This expands the select functionality to extend beyond the current page and include all filtered jobs, up to a configurable limit. The limit defaults to 1000 and may be overridden with a resolver callback.

:identification_card: Licensing and Installation

Oban Web v2.11 is licensed under Apache 2.0, just like Oban and Elixir itself. Previous versions are commercially licensed, therefore private, and won’t be published to Hex.

See the updated, much slimmer, installation guide to get started.

Oban Met v1.0

Oban Met is the secret sauce that powers the charts and runtime details shown in the Oban Web dashboard. It is a distributed, compacting, multidimensional, telemetry-powered time series datastore (Zang!) for Oban that requires no configuring. It gathers data for queues, job counts, execution metrics, active crontabs, historic metrics, and more.

Web is virtually useless without Met, so we’ve open sourced it as well :sparkles:!

While Met is designed for use with Web, it’s still mighty useful on its own for accessing runtime metrics and counts. Take a look at the docs for some useful examples.

Simplified Plans

As of today, Web and Web+Pro subscriptions are no longer available. Only Oban Pro.

Existing web subscriptions and license keys will continue working until you upgrade Web to v2.11+. However, you’ll continue paying for your subscription until you’re ready to upgrade (those are private packages that we’re serving still).

Legacy web customers will receive an email with an exclusive coupon to upgrade to Pro. Use this to continue receiving support and to benefit from lush features such as: workflows, chunks, decorators, structured args, hooks, global limits, dynamic plugins, and much more.

:heart_on_fire:Special thanks to all of our customers :heart_on_fire: that supported Oban and its ecosystem for the past five years. You are integral to us making open source Oban, and now Web, possible.

69 Likes

Amazing! Congratulations and thank you very much. Oban Web being open-sourced is you pretty much doing selfless charity and I give you a deep bow or it. :bowing_man:


Some errata, if you don’t mind.

oy → boy

Link domain here should go from hexdocs.om to hexdocs.pm.


Again thank you for this! I am super interested in Oban Web, Oban Met and SQLite support and will inspect them with interest.

2 Likes

Huh. Well, I guess this is the push I needed to actually try out oban. Nice work.

Having some installation issues. Firstly, I needed to update igniter:

% mix igniter.install oban
** (Mix) The task "igniter.install" could not be found. Did you mean "escript.uninstall"?
% mix archive.install hex igniter_new
[...installing]

But then I got another failure:

% mix igniter.install oban
checking for igniter in project ✔
** (UndefinedFunctionError) function Mix.Tasks.Igniter.New.add_igniter_dep/2 is undefined (module Mix.Tasks.Igniter.New is not available)

Maybe a few tweaks needed to the onboarding journey.

update: apparently some other lib had an old version of igniter in my deps. I forced it to 0.5.6 and now get this:

% mix igniter.install oban
checking for igniter in project ✔
** (UndefinedFunctionError) function :http_util.timestamp/0 is undefined (module :http_util is not available)
    (inets 9.3.1) :http_util.timestamp()
    (inets 9.3.1) httpc.erl:1158: :httpc.handle_request/9
    (igniter 0.5.8) lib/igniter/project/deps.ex:276: Igniter.Project.Deps.determine_dep_type_and_version/1
    (igniter 0.5.8) lib/igniter/util/install.ex:29: anonymous fn/1 in Igniter.Util.Install.install/4
    (elixir 1.18.1) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
    (igniter 0.5.8) lib/igniter/util/install.ex:28: Igniter.Util.Install.install/4
    (mix 1.18.1) lib/mix/task.ex:495: anonymous fn/3 in Mix.Task.run_task/5
    (mix 1.18.1) lib/mix/cli.ex:107: Mix.CLI.run_task/2

Feeling like igniter is not quite “done”…

edit 2: OK, let’s do it semi automatically

* Getting oban (Hex package)
(base) james@black-lagoon ele % mix oban.install
==> oban
Compiling 61 files (.ex)
Generated oban app
==> ele
The task 'oban.install' requires igniter. Please install igniter and try again.

For more information, see: https://hexdocs.pm/igniter/readme.html#installation

I added igniter to mix.exs

(base) james@black-lagoon ele % mix deps | grep igniter
* igniter 0.5.8 (Hex package) (mix)
  locked at 0.5.8 (igniter) fef19832
(base) james@black-lagoon ele % 
% mix oban.install
The task 'oban.install' requires igniter. Please install igniter and try again.

For more information, see: https://hexdocs.pm/igniter/readme.html#installation

I’m not having much luck here!

Final update: I just installed it manually, which wasn’t hard at all and took less time than typing the above. Skip the auto install!

:wave: @jtippett

We’ve addressed the issue and appreciate your comments, reports and attempts. The manual install guide is still there for anyone who doesn’t want to use Igniter.

We believe Igniter is the future of installs for Elixir. It will have some growing pains and need some polish before it’s ready for everyone.

10 Likes

Thanks a lot for this release and announcement. Being able to just “drop in” oban web yesterday evening in a nonprofit website while removing a bespoke limited equivalent is invaluable.

6 Likes

Congratulations Shannon and Parker :tada: and thank you for open sourcing Oban Web - I’m sure sure this will make a lot of people happy! :smiley:

It’s been great seeing Oban grow over the years and you building a successful business by offering a pro service - I am sure you are an inspiration to many and I can’t wait to see what’s next for you and Oban!

8 Likes

Terrific! Eager to learn more about Met!

Sorry to hear about this! These are new issues I haven’t seen before :frowning: We’re using :http_util internally but apparently that isn’t as available as I thought it was?

The other issues you faced should essentially be non-issues moving forward. Hex packages used to have a dependency on igniter, but now they express those only as optional dependencies, and your own project has to specify a dependency on igniter. Not that it makes your experience any better, but as long as people are using the latest versions of igniter, that kind of weird incompatibility/breakage shouldn’t be an issue moving forward.

So that leaves us with the :http_util issue. It looks like we were relying on that being available but that application may or may not be included in the host application.

The original reasoning behind using it in the first place was slimming down the deps, but now that igniter is listed as a dev/test dependency in end user’s applications, I’m just going to switch it to depend on req and use that. Its a lot easier to use anyway :slight_smile:

Thanks for walking through all the things that didn’t work, its super valuable :bowing_man:

2 Likes

The latest version of igniter no longer uses http_util/httpc. It adds a new dependencies (optional, meaning not deployed to production unless you have some other thing that needs req in prod), but is worth it for the simplicity and for the fixed behavior :slight_smile:

6 Likes

Great stuff, thank you and congratulations on the launch!

This part of the installation guide is out of date. It says {:oban_met, "~> 0.1", repo: :oban} which doesn’t work. I replaced it with {:oban_met, "~> 1.0"} and everything works as expected :yum:

1 Like

Very exciting! Did the price of just Oban pro increase with this change? I don’t see a mention of it so I could be wrong but I recall it being $99/mo with a yearly subscription :thinking:

Hi-ya,

Thank you! We updated and published the docs.

:confetti_ball:

It feels weird to ask but… does Oban Web have minimal system requirements? I have a pet project running on a very small flyio machine (shared-1x-cpu@1024MB) that does quite a bit of background processing with Oban (hundred of thousands of jobs, with SQLite to make things spicy). I upgraded to 2.19 and enabled Oban Web yesterday and suddenly the app is barely responsive. And that’s with good old dead view. Looking at Fly metrics, the CPU Utilization shows a lot of throttling - something I never seen before. Removing Web seems to help, but that’s about all the data I have at this point.

Have you seen this thread: CPU Quotas Update - Fresh Produce - Fly.io

Fly introduced CPU quotas. I’m not sure this is the thing you’re running into but you’re saying you see a lot of throttling so it could be this.

I haven’t seen this, thanks! That’s definitely good to know. Without Oban Web enabled, my app seems to be well below throttling level, so there’s that.