Graceful shutdown example from the Announcing Broadway @ Lambda Days 2019 talk


I just finished watching another inspiring presentation about upcoming Broadway library and the problems with complex pipelines that @josevalim talked about couldn’t omit my full attention.
That part start at 35:29

Most of the GenStage articles out there are describing scenario with a one single Producer. That is a gap in a knowledge.
I particularly never implemented scenario with multiple producers yet, but the time is coming :slight_smile:

If you guys ever done I would like to hear your experience, particularly about:

  • data processing locality - how do you ensure that events with a same foo_id are going through the same consumer?
  • graceful shutdown - that Terminator process magic that can delay the tree shutdown until all other process are done with their jobs.

Examples and samples will be much appreciated!


1 Like

GenStage.PartitionDispatcher will allow you to that as long as you hash/partition by foo_id.

I would recommend checking out the terminator in Broadway. The idea is to set the demand to accumulate and use async_info: