PartitionDispatcher: dynamic number of partitions

For a demo I am creating a Distributed GenStage system where each stage is in a different node. The basic architecture would be

A -> B -> C

So far so good, but I would also like to be able to scale each type of worker dynamically to multiple instances. This last requirement becomes a problem because I’d like to do a MapReduce kind of job (word counting) and need the PartitionDispatcher e.g. to dispatch events from n Bs to m Cs, however, since I don’t know the number of C workers in advance (since want to be able to scale C at any time) I cant set the number of partitions as required by this dispatcher.

Currently it is not possible to change the number of partitions. However, keep in mind that shrinking or expanding the number of partitions can have undesired results, as events were previously dispatched may now belong to partitions they were not supposed to. My advice would be to collect the information about the number of nodes available and start the main GenStage with the correct number of partitions (that’s what we do in Flow).