Separating API from background tasks (oban/quantum) for deployment

Hello Community,

I come from a predominantly python background and I am very new to elixir. So I am having a hard time figuring out some things in Elixir (Phoenix/Oban).

Recently, my company took over a Elixir product that someone else built. It was built with Phoenix and they have used Oban to run asynchronous tasks. These tasks are very compute intensive (they do a lot of calculations) and its slowing the APIs down.

If it was python, we usually have one command to run API server and another command to run background tasks (usually a celery command). So I build a separate “api” container and a separate “task” container. I just scale “task” container and it won’t slow down “api” containers as they run on independent machines.

I am having a hard time figuring out how to do the same thing with Pheonix/Oban. Basically, what I am looking for is a way to start API and Tasks separately (using two different commands/ passing some flags).

Plus I discovered they have used something called “libcluster” in the product (I have a basic idea. I know I might have to use a different clustering strategy when I do deployment). I am not sure if this is relevant, but I somehow felt that these things are connected.

Can someone please point me to relevant resources? I just want to be able to deploy API server & Oban (or quantum) tasks independently.

You probably need to run Oban in all application instances. Most likely the API inserts Oban jobs, which requires a running instance (though it needn’t run any queues).

You can configure each application instance to run specific queues and avoid heavy computation on API instances. There is an official guide that covers this exact use case: https://hexdocs.pm/oban/splitting-queues.html#content

5 Likes