Overview

For security compliance reasons, I need to ensure a complete separation between some of the consumers, and the producers that submit jobs to the queue. It’s sufficient to have the jobs go to a common PostgreSQL instance, but the Erlang VMs must be separate, to provide a strong security boundary.

In the past I’ve used RabbitMQ very successfully, but we don’t have a message broker yet in this project, and Oban’s workflows will be very useful for other aspects of the project.

Limitations

This rules out Erlang distributions obviously, and also PG Bouncer, but this is still enough to run Oban successfully using PostgreSQL LISTEN/NOTIFY. I asked this question on Slack, and got enough of a feel to implement this, so far it seems to work well enough.

Questions

• have I missed anything obvious from the proposed solution?
• has anybody been down this path, and has some war stories to share?
• are there any other patterns to deal with this situation?
• what’s the best way to notify the Producer that the work has completed? Note there’s no PubSub available between nodes. I found PostGrex.Notifications & Postgrex PubSub as well as Oban.Notifier

Current Solution

• a front end Phoenix app, with migrations, and Oban job submission

• a backend Elixir app, without migrations, but sharing Repo & Oban config

• a common Oban.Worker library module, used in both apps

• the primary has no queues defined via queues: []

• the secondary has the necessary queues defined queues: [default: 10]

• for notification the built-in Oban.Notifier is a better choice and works with the same underlying mechanisms.