I am looking for a data processing pipeline abstraction with which I can declaratively setup branching pipelines.
For now I have the following simple pipeline:
P --> PCa --> PCb --> C
'--> PCc ---^
P
is broadcasting to PCa
and PCc
.
I was not satisfied with Broadway in this context and modeled the pipeline with GenStage. Even with the simple example there are drawbacks to setting up GenStage. I would like to have everything setup and connected in one place with as little configuration needed outside of the supervisor initialization context.
Something like this:
Broadway2.start_link(__MODULE__,
name: __MODULE__,
pipeline: [
[Producer, [:start_link], :broadcast],
[PConsA, [:start_link], sub_to: [Producer], max_demand: 1],
[PConsB, [:start_link], sub_to: [PConsA]],
[PConsC, :start_link, sub_to: [Producer]],
[Consumer, :start_link,
sub_to: [PConsB, PConsC],
concurrency: 2,
rate_limiting: [
allowed_messages: 1,
interval: 1_000
]
]
],
)
Do you know any library I should take a look at?