Background job queues: When to use? When not to use? Which one to use?

I’m a Rihanna user and recently became a maintainer. For my use Postgres for persistence is the real winner as in my application I care little about performance and a lot about durability. I could attempt to manage the queue state within the cluster but I feel more confident entrusting this job to the fantastic piece of engineering that is Postgres. It also adds no additional operational overhead as I don’t need to form a cluster or add a dedicated external message queue (or Redis).

I’ve previously used Redis backed queues extensively and for me they hit an uncomfortable middle ground. They lack the durability guarantees of Postgres (or similar), they require me to deploy and maintain a Redis cluster, and lack the performance potential of working within the cluster.

I think your reasoning is correct here. If you need better assurances that you won’t loose jobs if a box is recycled or autoscaled away or otherwise disappears then it makes sense to write those jobs to some sort of storage. You can still run the job processing in your existing BEAMs if that makes sense for your setup.

Another valid use case you mentioned was sending messages between systems. This is a reasonable pattern, especially if you work in a polyglot system. This is a pattern we use at work with both rabbitmq and kafka. If you just need to send messages then rabbit is good. Kafka has really nice properties but its a chore and overkill for most companies IMO.

I haven’t used any of the job processing libs in elixir so I can’t speak to those. Generally though redis is very reasonable for these kinds of operations. I’d personally lean towards that over mnesia/dets. I don’t tend to reach for those tools because 1) My boxes typically have ephemeral file systems and I don’t feel like changing our ops setup and (much much less important) 2) there are some limitations around file sizes with dets and, as a byproduct, mnesia. I think klarna or someone built a leveldb engine for mnesia that looked interesting but I know nothing else about it. I’ve personally seen corruption issues in dets but that was a while ago and presumably that stuff is fixed. But it still made me nervous about those tools and I’ve never taken the time to build that confidence back up. But if you needed to you could presumably do all of this job processing with durable jobs, without ever leaving OTP which is interesting. It just hasn’t been something I’ve needed.

Thank you both for your insight @lpil and @keathley!

I love how @lpil put your choice pro rihanna - your app values the robustness higher than the throughput. Naturally what background queue to choose is heavily dependent on your requirement for guarantees of these systems as well as throughput etc.

I didn’t even think about “autoscaling” nodes away, as I’ve never worked in an autoscaling environment File Corruption is a good point, one of the reasons why I tend more to reach for more “robust” solutions such as Postgres or Redis

Thanks!

I think I tried to keep my personal experience a bit out of the initial post, I haven’t used a background queue in elixir (yet) but am otherwise a happy sidekiq (and hence redis) user when it comes to ruby land.

Could you expand on that, please? I was pondering using Kafka lately and I’m very interested in your reservations to it.

Shameless self-plug… But you may find it useful. I created GenQueue to handle the “background jobs beyond Task” case - allowing one to swap out libraries as needed. Also helps with testing. I personally use the TaskBunny and OPQ adapters alot. Allows the use of the same interface for both libraries which is nice.

added it to the list of queue systems, thanks for making us aware of it

I really like Kafka and have used it heavily for a few years now. But getting it running smoothly can be difficult depending on how familiar you are with ops and JVMs. At the end of the day you’re running a stateful service on a bunch of JVMs. So you’ll need a good understanding of how to get metrics out of the jvm and all of your boxes, you’ll wanna tune your jvms, you’ll want to tune your kafka setup, etc. The data in kafka can’t last forever so typically you window the available data for a limited time; generally no more than a month. So you’ll need a way to rotate the logs and shove them into s3 or some other long term storage. On top of all of that you most likely also need to run a zookeeper as well so you have to do all of that same ops work but this time for zookeeper.

If you’re just getting started I recommend that people vendor their kafka setup. Depending on how much scale you need it’ll probably run you somewhere between $500 and $3000 a month. Thats much cheaper then paying for a dedicated ops team so if its something you really need then it’ll be worth the expense IMO. But if you don’t need kafka’s properties (high write throughput and replicated, durable messages) then it may not be worth it.

Thanks a lot.

Would you be willing to tell us when you would opt for RabbitMQ and when for Kafka?

@dimitarvp my 2 cents since I researched the topic some time ago:

Kafka - dumb pipes and smart endpoints
RabbitMQ can have some extra logic like fancier routing

So you move complexity to the MQ or your app, the question is where it belongs, might depend. Also Kafka, being dumb, can handle more traffic, though the limit of them both is more then enough for most of the cases I chose RabbitMQ and was pretty happy but it really depends on the use case.

On Topic: I’d only use MQ to talk to external systems, otherwise GenServer + Tasks + Process.send_after is perfectly fine on the beam side. Postgres for persistance and then Ecto’s streaming to be able to process a lot of entries without exploding the RAM usage.

This is what we do at work, with rabbitmq (realtime pipeline of microservices processing data from different sources). I think it works really well for realtime data, when an event triggers a task/tasks in one or multiple microservices.

For batch processing.. I honestly prefer to use a system just for that, a system that doesn’t use messages but that executes one or multiple tasks, and gives me a log of what happened, something like the old good jenkins, just to say a name (which maybe is overkilling most of the time).
We also use kubernetes cronjobs for recurrent batch processing.. but still, I don’t feel as mush in control as having something like sidekiq, jenkins etc..

Anyone of you guys feel the same way on realtime vs batch?

PS: most of the codebase is in python, mainly because machine learning and data science/finance libraries