Evaluating Oban Pro for multiple nodes

Questions / Help
1

I am evaluating Oban Pro to see if it will meet my needs for job processing. Its seems like it checks all the boxes, but there are some things I am unsure about.

I need to be able to schedule jobs which will spawn n number of child jobs. The parent job has to wait for all the child jobs to complete. Seems like this is all do-able using Workflows.

What I am not sure about is what the requirements are for running in multiple nodes. In order for the workflow to ‘work’, do the jobs have to be in the same node(I believe the answer is no). However, what constitutes multiple nodes? Can I just stand up multiple instances of the app connecting to the same Postgres database? Will I be able to run the parent jobs in node 1 and child jobs in nodes 2, 3, … This seems baked in and docs reference PubSub for making this all work. My app is spread across multiple instances but are not in a BEAM cluster.

instance 1: running Phoenix and main Oban queue
instance 2, running media Oban queue.

When instance one creates a media job that will run in instance 2, will it get the job notifications without having to cluster the instances?

1 Like
2

Oban and Workflows are designed to be multi node, without any extra configuration.

PubSub is used in a plethora of places, (canceling jobs or staging jobs). However, it’s not required for Workflows. If it were required, Oban uses a Postgres notifier by default and that works in most environments.

3 Likes
3

When you mention node, what comes to my mind is a ‘node in a cluster’. I don’t think that is what you mean. Just want to make sure I am understanding you correctly. I think what Oban considers a node is just an instance of application that shares same Repo as another ‘node’. Am I correct in that?

What is “PubSub” based on? Is this Phoenix PubSub, or some other mechanism?

4

While you can insert jobs from any application and any language, oban nodes are generally erlang nodes as well.

It’s based on postgres listen/notify so you do not need to form a cluster with those erlang nodes.

5

Oban’s version of a node is the same as Erlang’s version of a node. Any application instance, regardless of if they are clustered. Oban’s been designed to work multi-node, even when they are not clustered. A surprising number of systems, (surprising to me)… aren’t clustered.

It’s not based on Phoenix because there is no dependency on Phoenix. It has two custom notifier implementations, one through Postgres and another based on pg for clustered nodes.