Can Distributed GenStage Manage Demand Globally?

Questions / Help
1

Here’s my situation: I have a pipeline for events I’d like to use GenStage to process. One stage in the pipeline makes calls to an external API with a simultaneous request limit, say 10. Creating a pipeline on one machine for this is simple, one producer and 10 consumers.

Now, for resiliency, we deploy our application to another machine and connect the nodes (AWS EC2 instances are ephemeral). The problem is now I have 20 consumers and I exceed my simultaneous request limit with the API.

Is there a way to adapter the number of consumers to a global max_demand?

I found this cool example of consumers running on multiple nodes and subscribing to a global producer. :icon_cool:

2

The super simple version is to just do

local_demand = total_demand / (length(Node.list))

You could totally try to have demand moved between nodes but this way the there’s no need to coordinate between nodes. If however you add more nodes part way through you have to restart stuff.

1 Like