bluzky

bluzky

Should I use Broadway or exq or both?

Hi,

I’m building a system which pull lots of order, chat message, and other data from 3rd services then process these data. I intend to use Broadway to process data after pulling to my system, and use exq to run background job.

I think that basically they use queue to store message/job and then pick from queue to process message/job. I want to make my system simple so I am thinking of using 1 solution for both.

Could you give me advice if I should use broadway for background job processing and when to use broadway and when to use job queue like exq?

Thanks.

Most Liked

stefanchrobot

stefanchrobot

I think the decision is going to be mostly based on the features that you’re after which I think should be connected with the amount of “work items”/jobs that you want to process. While people usually use Broadway to consume data from AWS SQS, RabbitMQ or Kafka, nothing is stoping you from writing your own producer. On the other hand, job processing libraries don’t really allow for customization of the job storage, but they usually come with some sort of UI for inspecting and managing the jobs.

Consider also the complexity of the infrastructure. If I have an app that pulls data from APIs and uses PostgreSQL, I’d use Oban for background jobs (which has the sweet benefit of scheduling jobs in the same DB transaction as updating the data). On the other hand, if I’m processing data from an AWS SQS queue, I might as well create a different SQS queue for background jobs (latency could be a factor here though) and use Broadway for that.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43757 214
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement