Should I use Broadway or exq or both?


I’m building a system which pull lots of order, chat message, and other data from 3rd services then process these data. I intend to use Broadway to process data after pulling to my system, and use exq to run background job.

I think that basically they use queue to store message/job and then pick from queue to process message/job. I want to make my system simple so I am thinking of using 1 solution for both.

Could you give me advice if I should use broadway for background job processing and when to use broadway and when to use job queue like exq?


I think the decision is going to be mostly based on the features that you’re after which I think should be connected with the amount of “work items”/jobs that you want to process. While people usually use Broadway to consume data from AWS SQS, RabbitMQ or Kafka, nothing is stoping you from writing your own producer. On the other hand, job processing libraries don’t really allow for customization of the job storage, but they usually come with some sort of UI for inspecting and managing the jobs.

Consider also the complexity of the infrastructure. If I have an app that pulls data from APIs and uses PostgreSQL, I’d use Oban for background jobs (which has the sweet benefit of scheduling jobs in the same DB transaction as updating the data). On the other hand, if I’m processing data from an AWS SQS queue, I might as well create a different SQS queue for background jobs (latency could be a factor here though) and use Broadway for that.