GenStage: Difference between ProducerConsumer and ConsumerSupervisor - modelling different levels of concurrency to several external APIs


I’m trying to model a problem where I have to access 2 external APIs and they provide different levels of concurrency, let’s call them “q” (1 request at a time) and “av” (2 concurrent requests).

I implemented an initial version which looks like this:

            \---- [C]

The Producer is implemented using GenStage.BroadcastDispatcher and the ProducerConsumers subscribe only to events of a certain type (:q, or :av)

This works as expected, with a maximum of 3 concurrent requests (1 for q, 2 for av).

I then tried to replace the ProduceConsumers and the Consumers with a ConsumerSupervisor which seems like an even better approach, less code and simpler conceptually.

I can’t figure out how to get this version to work. I’d expect to be able to tune the concurrency from the ConsumerSupervisor subscription to the Producer with max_demand and it seems to work but I don’t understand why :q and :av ConsumerSupervisors aren’t working at the same time.

I’ve put the code in a gist together with the outputs of both versions, please let me know what I’m doing wrong!


1 Like