bluzky
Hi,
I’m building a system which pull lots of order, chat message, and other data from 3rd services then process these data. I intend to use Broadway to process data after pulling to my system, and use exq to run background job.
I think that basically they use queue to store message/job and then pick from queue to process message/job. I want to make my system simple so I am thinking of using 1 solution for both.
Could you give me advice if I should use broadway for background job processing and when to use broadway and when to use job queue like exq?
Thanks.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
stefanchrobot
I think the decision is going to be mostly based on the features that you’re after which I think should be connected with the amount of “work items”/jobs that you want to process. While people usually use Broadway to consume data from AWS SQS, RabbitMQ or Kafka, nothing is stoping you from writing your own producer. On the other hand, job processing libraries don’t really allow for customization of the job storage, but they usually come with some sort of UI for inspecting and managing the jobs.
Consider also the complexity of the infrastructure. If I have an app that pulls data from APIs and uses PostgreSQL, I’d use Oban for background jobs (which has the sweet benefit of scheduling jobs in the same DB transaction as updating the data). On the other hand, if I’m processing data from an AWS SQS queue, I might as well create a different SQS queue for background jobs (latency could be a factor here though) and use Broadway for that.