Parallel job processing in Oban

Questions / Help
, , ,
1

I have created a queue in Oban with a concurrency of 12. The worker’s job is to take the parameters, make an http call and insert the response into database. However when I inserted jobs using Oban.insert_all, I see that the jobs are getting processed serially and not in parallel. Am I missing any config ? Or is it something provided in Oban pro.

2

is the http library running a queue of 1?

else show some pseudo code, and/or debug by removing http call and do something like this in the worker:

IO.inspect("job started")
:timer.sleep(5_000)
IO.inspect("job done")

and see if they are parallel…

1 Like
3

If you have 12 workers, they should all try to process a job, which means you should see 12 jobs in progress at once (parallel).

Can you tell us how you are sure they’re happening serially?

1 Like
4

@outlog @benwilson512 So I tried to simulate using the above snippet as well.

IO.inspect("job started")
:timer.sleep(5_000)
IO.inspect("job done")

The output is serial. After every 5 seconds, Job Done gets printed. If it were parallel, I should have seen at least 12 get printed together.

My config:

config :App, Oban,
  repo:App.Repo,
  plugins: [Oban.Plugins.Pruner],
  queues: [test: 12],
  testing: :inline

Code:

def perform(params) do
    IO.inspect("job started")
    :timer.sleep(5_000)
    IO.inspect("job done")
end
1 Like
5

Ah. This is because you’re running with inline test mode. That should only be enabled in test.exs, not for your base config.

The way you have it configured you’re never persisting anything into the database, it’s executing them one at a time in memory.

5 Likes
6

Thank you so much. It’s working now. It’s been more than a day. :sweat_smile:

3 Likes