Here’s my situation: I have a pipeline for events I’d like to use GenStage to process. One stage in the pipeline makes calls to an external API with a simultaneous request limit, say 10. Creating a pipeline on one machine for this is simple, one producer and 10 consumers.

Now, for resiliency, we deploy our application to another machine and connect the nodes (AWS EC2 instances are ephemeral). The problem is now I have 20 consumers and I exceed my simultaneous request limit with the API.

Is there a way to adapter the number of consumers to a global max_demand?

I found this cool example of consumers running on multiple nodes and subscribing to a global producer.