Help with tuning data flow in Broadway: c:prepare_messages/2 question

I don’t think the docs are particularly helpful here. Yes, min_demand/max_demand configure how many things processors request from producers. That however doesn’t also mean that the producer will emit produced events/messages in batches related to those settings. By my understanding the lists prepare_messages receives map to the lists emitted by individual GenStage callbacks being called. If those callbacks only emit list of individual events then prepare_changes will receive only such.

If you want to batch those up you’d likely need a batcher (e.g. genstage consumer_producer) between your current producer and broadway, which aggregates individual events into batches. Or see if your producer could be configured to emit events in batches.

Edit:

There it is for rabbitmq:
https://github.com/dashbitco/broadway_rabbitmq/blob/f4682ebcdc8c3bbb241004f277f5998a95fddf77/lib/broadway_rabbitmq/producer.ex#L533

Ahhhh thank you! I don’t feel so much like I’m going crazy now. For my mental model it’s surprising that prepare_messages/2 isn’t aggregating, given that it’s receiving a list. But hey, there it is.

It makes sense due to how GenStage works, but it is indeed strange given how this is marketed to Broadway users.