rump13
Combining Broadway and Oban for SQS heavy message processing
Hello everyone! I’m new to Elixir and this is my first post here.
I’m currently developing a web application that requires processing of messages from a single AWS SQS queue. The queue contains two types of messages: fast messages that need to be processed quickly, and slow messages that can wait. The fast messages make up 95% of the queue, while the slow messages are only 5%. Processing the fast messages is simple: I just need to extract some data from the notifications and store them in a database. On the other hand, processing the slow messages requires a lot of computation and IO resources, but it doesn’t need to be done in almost-realtime.
Currently, I’m using BroadwaySQS.Producer to receive messages from AWS SQS and process the fast messages. For the slow messages, I plan to receive them in the same Broadway pipeline and create a job for each one using Oban. This way, the slow messages can be delegated to Oban as jobs that can be processed later. I’ll be using the free version of Oban.
Do you see any issues with integrating Broadway pipelines with Oban jobs? Or is it possible to solve this problem using only Broadway?
Happy coding! 
Marked As Solved
tmartin8080
You could likely use a separate SQS queue and broadway pipeline to process fast/slow messages.
- SQS:
fast.queue,slow.queue - Broadway:
FastPipeline,SlowPipeline
Then either publish messages from current publisher or from FastBroadway as it comes across a “slow” message.
If it helps, you can control the concurrency for processors etc.
Also Liked
sorentwo
@rump13 your plan seems sound to me. Broadway is meant for ingesting messages with back pressure, not as a general purpose background job system.
@kanishka this article explains why you’d want to use Oban instead of plain tasks: Oban Starts Where Tasks End · Oban Pro
tmartin8080
those slow messages/jobs don’t get lost
Might also be worth mentioning that Broadway handles exceptions during processing and passes any failed messages to handle_failed/2 so you can log/alert/cleanup/re-publish etc. SQS messages won’t be deleted until they’re acknowledged so there’s a low chance of them getting lost.
We use RabbitMQ with Broadway and create dynamic queues for retries and exponential backoff. It seems like SQS supports delayed queues but haven’t used them.
Oban is definitely an option (and an amazing tool thanks @sorentwo 
) but might be worth exploring what’s possible with your current setup.
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"