I’ve got an SQS message which triggers a large postgres query, the results of which are then processed in batches, performing a postgres update for each batch. I’m looking at using Broadway for the SQS integration and GitHub - mtwilliams/bourne: 🚤 Better streaming for Ecto. · GitHub which allows you to use a postgres streaming query as a gen_stage producer.

Where I’m a little fuzzy is how to integrate Broadway and Bourne. In particular Bourne creates a new stage each time you run a postgres query. This would mean that the handle_message (or perhaps handle_batch) callback would create a new gen_stage producer for the postgres query, effectively creating a Bourne gen_stage pipeline inside the Broadway gen_stage pipeline.

Should the Bourne pipeline run inside a single stage inside Broadway? Another option might be for Broadway to delegate to the Bourne pipeline when fetching demand?