I am trying to wrap my head around the difference between Oban & RabbitMQ i.e. when does it make sense to use Oban vs RabbitMQ?
I know RabbitMQ is one extra component in your infrastructure requiring special config, monitoring etc so from that perspective Oban is easier to setup/monitor. But in terms of support for specific use cases or features which one would you choose and why?
They don’t really compete in the same space…
Oban is kinda job scheduler, backed by database.
RabbitMQ is kinda Enterprise Bus, where You submit and listen to events.
But could Oban be used for submitting and listening for events like RabbitMQ? and conversely can RabbitMQ be used as job scheduler?
It seems RabbitMQ will be well suited when integrating 2 or more applications through a common bus using events, right?
Not really, You need a rabbitmq client. You use Oban to start a job each night for example.
Right
In terms of performance at what point in time does Oban/Postgres show its limitations to warrant using RabbitMQ?
Oban is not RabbitMQ, I don’t see one replacing the other.
RabbitMQ is purely a message broker, which is suitable for sending persistent messages between nodes. It doesn’t handle scheduling, retries, concurrency, testing concerns, or any of the other features Oban provides. To get some of those features you’ll need to use a library like Broadway, but it still won’t handle errors/retries/schedules/discards or provide the ability to query your data.
Regarding the performance, Oban can easily process 1-10k jobs per-queue per-node, depending on your concurrency levels.
in what unit of time?
That is per second, as measured on my local machine.
@sorentwo Since you are the author of Oban, is it possible for you to share benchmarks since 1-10 is pretty wide range and I need to justify to my superiors in a Forture 200 company the choice of using Oban vs RabbitMQ, which I understand is not the right choice for job scheduling and requires jumping hoops and managing retries etc.
Some benchmarks are included in the Oban repo. The variance is down to two factors:
With a cool down of 1ms and a concurrency of 1, you can run at most 1k in a second. With a more realistic concurrency of 10 you can run 10k in a minute.
Most of my benchmarking focuses on throughput, or how many jobs can be processed with sustained inserts and execution simultaneously.
If you just need a job processing and are already using PostgreSQl, I highly recommend using Oban. Message queue/broker, such as Kafka, RabbitMQ etc. can be used for job queueing but your application needs to handle complicated job specific features since they don’t have certain features handy (such as locking). There are many small gotchas you may hit if you write own job processing with job queue, especially ACID is required.
Message queue/broker have its own use cases - if you’re not sure probably you don’t need it 
(I’m using Oban in two production services for low-volume but important jobs)
Oban is good, but we noticed we creeped up its usage without really trying by just throwing random things into it, so it is something that needs to be considered carefully. The use cases for oban and a message queue are very different and you must consider them carefully.
We have 8 nodes that are clustered and having oban on each one means that they really slam our database staying in sync. The nodes could just communicate their state in vm very cheaply but they use the db since that is how oban works.
We found ourselves doing things like creating unique jobs with delays to prevent jobs operating on the same data at the same time on different nodes and that was a sign that we were not using the right tool. In a message queue messages are ordered and they can be hashed so that messages that are related to each other always happen on the same node.
Oban will run through its jobs and if one fails, it will retry, Great. But they won’t necessarily happen in order. A message queue gives some guarantee on ordering. Often you can make it so that if there is a failure it will just keep retrying that message instead of skipping past. Then when you fix the bug, the process just continues right where it left off.
Oban (at least our version) has job limits but they appear to be capped on a per node level, so if we say create a job that has a limit of one, but we have eight nodes, there will be eight of them running at a time. Message queues often have partitioned topics and that prevents more nodes from operating it than you intended.
Oban is great, but remember what its use case is.
You have some great points in here. Oban is a background job processor, not a message queue or a kafka replacement.
Just to note, this changed completely in Oban 2.4, as now individual queues don’t poll the database at all. Instead, a single process manages scheduled jobs and uses notifications to notify queues that they have jobs to run.
That’s good to know! One of these days when we can get some breathing space we are going to get upgraded to a more recent version, as I’m sure there are a lot of improvements we are missing out on.
© Copyright Elixir Forum | Terms | Privacy & Cookies
