Help with tuning data flow in Broadway: c:prepare_messages/2 question

I don’t think the docs are particularly helpful here. Yes, min_demand/max_demand configure how many things processors request from producers. That however doesn’t also mean that the producer will emit produced events/messages in batches related to those settings. By my understanding the lists prepare_messages receives map to the lists emitted by individual GenStage callbacks being called. If those callbacks only emit list of individual events then prepare_changes will receive only such.

If you want to batch those up you’d likely need a batcher (e.g. genstage consumer_producer) between your current producer and broadway, which aggregates individual events into batches. Or see if your producer could be configured to emit events in batches.

Edit:

There it is for rabbitmq:

1 Like