Graceful shutdown example from the Announcing Broadway @ Lambda Days 2019 talk

Questions / Help
1

Hi,

I just finished watching another inspiring presentation about upcoming Broadway library and the problems with complex pipelines that @josevalim talked about couldn’t omit my full attention.
That part start at 35:29 https://www.youtube.com/watch?v=Fq9hpl7xbWM&t=35m29s

Most of the GenStage articles out there are describing scenario with a one single Producer. That is a gap in a knowledge.
I particularly never implemented scenario with multiple producers yet, but the time is coming :slight_smile:

If you guys ever done I would like to hear your experience, particularly about:

  • data processing locality - how do you ensure that events with a same foo_id are going through the same consumer?
  • graceful shutdown - that Terminator process magic that can delay the tree shutdown until all other process are done with their jobs.

Examples and samples will be much appreciated!

:v:

1 Like
2

GenStage.PartitionDispatcher will allow you to that as long as you hash/partition by foo_id.

I would recommend checking out the terminator in Broadway. The idea is to set the demand to accumulate and use async_info: https://github.com/plataformatec/broadway/blob/master/lib/broadway/terminator.ex

3

Thanks!