Hi everyone, I’ve got a question related to project architecture.

In the microservice sphere there is this concept where you can scale up individual services when needed. So imagine the following scenario:

     ┌────────────────────┐
     │RabbitMQ Message Bus│
     └────────────────────┘
       ▲            ▲
       │            │
 ┌─────┴───┐    ┌───┴──────────┐
 │Webserver│    │Expensive jobs│
 └─────────┘    └──────────────┘

The precise job doesn’t really matter, but let’s assume that we’re processing and/or modifying images or something.
The ‘expensive jobs’ box retrieves jobs from RabbitMQ and returns the result.
Let’s say that the expensive jobs get popular with our users, so we want to add another box that retrieves jobs from RabbitMQ.

     ┌────────────────────┐
     │RabbitMQ Message Bus│
     └────────────────────┘
       ▲            ▲   ▲
       │            │   └──────────┐
       │            │              │
 ┌─────┴───┐    ┌───┴──────────┐ ┌─┴────────────┐
 │Webserver│    │Expensive jobs│ │Expensive jobs│
 └─────────┘    └──────────────┘ └──────────────┘

We’ve now effectively doubled our capacity for handling these expensive jobs.

My question is as follows: how would you achieve something similar in an Elixir (Phoenix) monolith? We can use Oban to perform background jobs, but what if the amount of jobs becomes sufficiently large that the queue reaches its limit. We could theoretically scale up the entire server, but that doesn’t feel correct for some reason.