Something to remember when you are developing in the BEAM: The scheduler is preemptive, operating on a time-sharing principle. This means that the VM will actually pause long-running processes (such as your background jobs) from time to time in order to let other processes (such as your web requests, which should be very short-lived) have a turn. This is done by allocating a specific number of reductions (function calls) to each process, after which the scheduler will switch to the next process in the queue.

The primary advantage of this approach is that it ensures every process gets some CPU time even under very heavy load, promoting fairness and responsiveness. It is not to say that your background jobs will have no negative impact on the time it takes to finish processing a request, but it’s not nearly as detrimental as a cooperative scheduler, which would not pause processes and just lets them run until they finish what they are doing. This can lead to issues like process monopolization, where a single process takes over the CPU and doesn’t allow other processes to run.

In Erlang’s system, the preemptive scheduling helps in distributing resources evenly, maintaining system responsiveness, and avoiding the risk of any single process overwhelming the system. This design reflects Erlang’s focus on concurrency, fault tolerance, and distributed computing, making it particularly suitable for scalable and highly available systems.

I get where you are coming from and I am a paranoid prepper myself but I’d advise you to measure first before going into a potential rabbit hole.

We run our Oban workers on separate VMs (Heroku dynos) that don’t serve web traffic. While the BEAM scheduler is great, it’s useful having a separate CPU/memory environment for each that can be scaled independently.

We do this by disabling the queues on the web workers with an environment variable which is read by the runtime.exs config.