bluzky

bluzky

Should I use Broadway or exq or both?

Hi,

I’m building a system which pull lots of order, chat message, and other data from 3rd services then process these data. I intend to use Broadway to process data after pulling to my system, and use exq to run background job.

I think that basically they use queue to store message/job and then pick from queue to process message/job. I want to make my system simple so I am thinking of using 1 solution for both.

Could you give me advice if I should use broadway for background job processing and when to use broadway and when to use job queue like exq?

Thanks.

Most Liked

stefanchrobot

stefanchrobot

I think the decision is going to be mostly based on the features that you’re after which I think should be connected with the amount of “work items”/jobs that you want to process. While people usually use Broadway to consume data from AWS SQS, RabbitMQ or Kafka, nothing is stoping you from writing your own producer. On the other hand, job processing libraries don’t really allow for customization of the job storage, but they usually come with some sort of UI for inspecting and managing the jobs.

Consider also the complexity of the infrastructure. If I have an app that pulls data from APIs and uses PostgreSQL, I’d use Oban for background jobs (which has the sweet benefit of scheduling jobs in the same DB transaction as updating the data). On the other hand, if I’m processing data from an AWS SQS queue, I might as well create a different SQS queue for background jobs (latency could be a factor here though) and use Broadway for that.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

Other popular topics Top

TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41989 114
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement