mgwidmann

mgwidmann

Broadway Message Distribution Strategy

Here is my system loaded up with work:

https://imgur.com/a/RWN6G4d

It seems Broadway is favoring several workers and giving them lots of messages to process (when I have max_demand: 1). Others are getting no messages, so turning up the number of workers has no effect on throughput. CPU & Network never hit maximum, the system just processes slowly.

Does Broadway use a round-robin message distribution strategy? Can I change it? How would I ensure each worker only ever has 1 message to work on at a time?

Most Liked

josevalim

josevalim

Creator of Elixir

It uses the default demand dispatcher from GenStage. Can you please post your topology definition? A high number of workers and low max_demand should be enough to make it process one by one. Also, you may be running into the issue that the producer cannot produce fast enough, so there is not enough work for all cores. More information would be appreciated.

josevalim

josevalim

Creator of Elixir

Yes, apparently I can’t internet today.

josevalim

josevalim

Creator of Elixir

Apologies, I only check the forum a couple times per week. Let’s move this to a Broadway issue and discuss there. But one issue is for sure: if you have multiple producers, the first processor would subscribe to all of them and be the first in the list to receive messages from all of them. The minimum we can do here is to shuffle the list of subscription. I have some ideas to try out, please open up an issue on Broadway and we can continue the convo there!

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New

We're in Beta

About us Mission Statement