Correctly setting up Oban (Pro) with "only" config

Hi, we’re using oban pro 1.5.0-rc.4, I checked the changelog for the later versions but I don’t think I see anything related to this issue.

Before going into detail (just to make sure I don’t miss the point), I have an issue and a question. The gist of the issue is that Oban doesn’t appear to respect the only config, which I will elaborate more on below. My question is whether we’re setting this up correctly, and how to better set Oban up for multiple users running the same env.

We test our features on our staging environment by running it locally on terminals, and each laptop is picked up as a node. We typically run this config only: {:sys_env, "DO_SCHEDULER", "do_scheduler"} and have that key-value pair saved on our laptops.

This presents the issue where if we have a terminal running staging env on our laptops at the same time, the jobs go to each other’s laptops. We tried to prevent this issue by changing the key on my laptop to only: {:sys_env, "EX_LOCAL", "local"}, but the next person who starts up their app will override that only config in the oban_queues table (I’m assuming that’s where the config is saved).

However, I noticed another issue where the oban_queues config has only: {:sys_env, "EX_LOCAL", "local"} saved, but a colleague who has DO_SCHEDULER saved on their laptop ended up picking up my jobs instead.

I tried changing the config to only: {:node, :=~, "<my name>"} because our laptops are shown as <name>-Macbook-Pro-<uuid> in the attempted_by column. I tested this with my colleague who had the original only: {:sys_env, "DO_SCHEDULER", "do_scheduler"} config and made sure oban_queues showed only: {:node, :=~, "<my name>"}, but the jobs were also being picked up by their laptop.

So I’m not sure if we’re going about this correctly in the first place? How can we better improve our setup to prevent jobs from being picked up by each other’s laptops?

Thank you.

Putting aside my confusion at sharing a staging environment between multiple people’s laptops…I’ll try and address how to work around the problem.

DynamicQueues is designed to work in a single environment with heterogeneous nodes. As you’ve discovered, the only options are persisted in the oban_queues table and the last-write will win. With a centralized table, a better option is to start separate queues for each user and use an environment variable on the dev machine to choose which queue jobs go into.

Insert a queue for each user in the dev/staging environment:

queues: [
  bob: 10,
  sue: 10,
  joe: 10
]

Set a queue override in runtime.exs:

config :my_app, :oban_queue_override, System.get_env("OBAN_QUEUE_OVERRIDE", "default")

Then, use that option to set the queue when inserting jobs:

queue = Application.fetch_env!(:my_app, :oban_queue_override)

args
|> MyApp.Worker.new(queue: queue)
|> Oban.insert()
4 Likes

Putting aside my confusion at sharing a staging environment between multiple people’s laptops…I’ll try and address how to work around the problem.

I have no words for this :smile:

Thanks for the explanation and proposed solution! I’ll discuss this with the others in my team, obviously the better long-term solution would be to… improve the way we handle our staging environment :joy:

1 Like