Run all background jobs in a different server

There are a few questions that need to be answered. What does “a decently high rate of api calls” mean? Do the API request handling put high load on the CPU relative to RAM? It usually doesn’t but history has seen many different cases. Maybe a load/stress test can show how much requests and background tasks your service can take?

There are 2 ways to go about this:

1. Rate limit the processing - use the Oban rate limiter (not sure how it works and it seems you need Oban pro) or Broadway or Genstage, this will save a lot of complexity and time in general.
2. Use RabbitMQ or built-in erlang message passing from node to node to pass events to another server that will do the processing. In this case you need to decide whether losing events is acceptable and a strategy of what happens when something goes down, much more complex compared to the first solution.

I get where you are coming from and I am a paranoid prepper myself but I’d advise you to measure first before going into a potential rabbit hole.

I agree that I may be getting to a potential Rabbit hole. I will leave things as is for now and try instead to add more metrics and limit background workers concurrency to a certain level.

Something to remember when you are developing in the BEAM: The scheduler is preemptive, operating on a time-sharing principle. This means that the VM will actually pause long-running processes (such as your background jobs) from time to time in order to let other processes (such as your web requests, which should be very short-lived) have a turn. This is done by allocating a specific number of reductions (function calls) to each process, after which the scheduler will switch to the next process in the queue.

The primary advantage of this approach is that it ensures every process gets some CPU time even under very heavy load, promoting fairness and responsiveness. It is not to say that your background jobs will have no negative impact on the time it takes to finish processing a request, but it’s not nearly as detrimental as a cooperative scheduler, which would not pause processes and just lets them run until they finish what they are doing. This can lead to issues like process monopolization, where a single process takes over the CPU and doesn’t allow other processes to run.

In Erlang’s system, the preemptive scheduling helps in distributing resources evenly, maintaining system responsiveness, and avoiding the risk of any single process overwhelming the system. This design reflects Erlang’s focus on concurrency, fault tolerance, and distributed computing, making it particularly suitable for scalable and highly available systems.

Yep, absolutely add metrics and/or telemetry.

GitHub - openobserve/openobserve: Open source observability platform for logs, metrics, traces, frontend monitoring, pipelines and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage costs and single binary deployment. · GitHub is a pretty good self-hosted solution. I already love it.

We run our Oban workers on separate VMs (Heroku dynos) that don’t serve web traffic. While the BEAM scheduler is great, it’s useful having a separate CPU/memory environment for each that can be scaled independently.

We do this by disabling the queues on the web workers with an environment variable which is read by the runtime.exs config.

Interesting approach. Do you have any kind of autoscale in place?
For example how do you scale your heroku dynos based on jobs in the queue.

I have not hear about it before. I will definitely try it.
What kind of metrics do you track?

No, our main bottleneck is our database and auto scaling up more dynos would just increase the load on this to the point where it had an effect on web users. Auto scaling down during quiet periods might save us some money, but it’s not a cost that’s meaningful right now.