For a demo I am creating a Distributed GenStage system where each stage
is in a different node. The basic architecture would be
A -> B -> C
So far so good, but I would also like to be able to scale each type of worker dynamically to multiple instances. This last requirement becomes a problem because I’d like to do a MapReduce kind of job (word counting) and need the PartitionDispatcher
e.g. to dispatch events from n
Bs to m
Cs, however, since I don’t know the number of C workers in advance (since want to be able to scale C at any time) I cant set the number of partitions as required by this dispatcher.