GenStage question: BroadcastDispatcher and automatic subscription

Hello all, I have some questions regarding usage of BroadcastDispatcher and the subscription pattern of multiple consumers.

With automatic subscription handling, ask method will be called whenever subscription is completed. The problem is when we have multiple consumers and we subscribe it one by one to the producer, the first consumer will trigger the producer’s handle_demand, although not all consumers have finished subscribing. If the events is fetched within handle_demand, there is possibility that the second consumer will not receive the events.

We could use some manual subscription to only ask for demand when all consumers have finished subscribing, but I think it’s quite cumbersome to track and count the demand after that.

I think it will be very helpful if GenStage have some ways to do “group subscription” for all consumers at once. What do guys think?

@alukito if you want to synchronize the starting of multiple consumers, you can use the demand option. When you start the producer, you should return demand: :accumulate on its init. Then you subscribe all children and you do a GenStage.demand/2 call setting it back to :forward.

Your supervision tree would look like this:

  • Producer
  • Consumer1
  • Consumer2
  • Consumer3
  • Temporary task that will change the demand

You want it to be one_for_all, so any crash causes everything to restart.

3 Likes

Ah, this is great. Thank you @josevalim, I just read the GenStage.demand/2 docs and I think it could work.