I am looking for a data processing pipeline abstraction with which I can declaratively setup branching pipelines.
For now I have the following simple pipeline:
P --> PCa --> PCb --> C '--> PCc ---^
P is broadcasting to
I was not satisfied with Broadway in this context and modeled the pipeline with GenStage. Even with the simple example there are drawbacks to setting up GenStage. I would like to have everything setup and connected in one place with as little configuration needed outside of the supervisor initialization context.
Something like this:
Broadway2.start_link(__MODULE__, name: __MODULE__, pipeline: [ [Producer, [:start_link], :broadcast], [PConsA, [:start_link], sub_to: [Producer], max_demand: 1], [PConsB, [:start_link], sub_to: [PConsA]], [PConsC, :start_link, sub_to: [Producer]], [Consumer, :start_link, sub_to: [PConsB, PConsC], concurrency: 2, rate_limiting: [ allowed_messages: 1, interval: 1_000 ] ] ], )
Do you know any library I should take a look at?