GenStage: Difference between ProducerConsumer and ConsumerSupervisor - modelling different levels of concurrency to several external APIs

Questions / Help
,
1

Hello!

I’m trying to model a problem where I have to access 2 external APIs and they provide different levels of concurrency, let’s call them “q” (1 request at a time) and “av” (2 concurrent requests).

I implemented an initial version which looks like this:

[P]----[PC]---[C]
  \               
    ----[PC]---[C]
            \---- [C]

The Producer is implemented using GenStage.BroadcastDispatcher and the ProducerConsumers subscribe only to events of a certain type (:q, or :av)

This works as expected, with a maximum of 3 concurrent requests (1 for q, 2 for av).

I then tried to replace the ProduceConsumers and the Consumers with a ConsumerSupervisor which seems like an even better approach, less code and simpler conceptually.

I can’t figure out how to get this version to work. I’d expect to be able to tune the concurrency from the ConsumerSupervisor subscription to the Producer with max_demand and it seems to work but I don’t understand why :q and :av ConsumerSupervisors aren’t working at the same time.

I’ve put the code in a gist together with the outputs of both versions, please let me know what I’m doing wrong!

Thanks!

https://gist.github.com/vicmargar/c28d59dda98602447aac5b17efc16774

1 Like