Run only one instance of Oban

Questions / Help
,
1

I have an Elixir apps which does some harder jobs in the background. I am using Oban to run them asynchrously. For better availability I’d like to spawn a second server instance of my Elixir app (I am using fly.io) but without spawing another Oban app there.

Currently Oban is just part of the Standard Application Supervision Tree:

def start(_type, _args) do
    children = [
      # Start the Ecto repository
      MyApp.Repo,
      MyAppWeb.Telemetry,
      {Phoenix.PubSub, name: MyApp.PubSub},
      MyAppWeb.Endpoint,
      {Oban, Application.fetch_env!(:my_app, Oban)}
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

Any idea how to do it?

2

You can:

  • Conditionally add Oban to the list of children (if ..., do: base_children, else: base_children ++ [{Oban, ...}]),
  • Conditionally tell Oban not to do any processing (config :my_app, Oban, queues: false, plugins: false).

You need to figure out the condition. This can be based on the value of node() or some ENV var passed during startup.

3

Can’t Oban work across multiple nodes and share the processing load?

1 Like
4

I guess it can. But I have like jobs which need a lot of processing power. So I don’t wanna let them run on my web facing machine which has less power.

1 Like
5

There is an official guide on this exact subject: Splitting Queues Between Nodes — Oban v2.10.1 :wink:

1 Like
6

I see, thank you. My problem is more that all of the machines have the same set of environment variables. So I need to figure out how to start it on the first machine only or have a different set of environment variables.

7

Interesting, are the FE and BE nodes setup in a distributed Erlang cluster?

8

The Fly way to do this, AFAIK, is to make separate apps that share a network and a database: Running Multiple Processes Inside A Fly.io App

1 Like
9

You might also consider using Highlander to ensure a process only runs once, globally, in a cluster. I use it for Quantum cron jobs.

2 Likes
10

Oban already ensures that jobs only run once for the given period across the entire cluster (even across restarts).

You could use leadership to only run those jobs in one node. However, then you can’t scale or control whether those nodes also serve web requests.

11

Sorry, didn’t mean to reply directly to you! Oban does a great job of this already, but if you want to run any old process as a singleton globally, Highlander is worth looking at.

12

This is only available as Oban Pro, isn’t it? Global Concurrency basically.

13

I believe that guarantee is available in non-paid version. Though you’d need to upgrade to Pro to have more control across the whole cluster (see SmartEngine).

14

Global concurrency requires Pro, but uniqueness (as used by cron) is available in all versions.

15

Just to understand that better. So with uniqueness you mean, the job is only executed once not multiple times when you have multiple nodes, right?
And Global Concurrency means that you can say, I want to run x jobs across all of my nodes. Without you could only set a concurrency limit per node and the global limit is basically: local_limit x number_of_nodes. Is that correct? Thank you!

1 Like
16

You summarized it nicely.

One small note regarding Oban’s cron and how it interplays with uniqueness: the cron plugin inserts a single job which may be picked up by any node, just like any other job.

1 Like