idoshamun
Multi stage pipelines with Broadway
I want to build a system that supports user provided multi steps workflows. Every such step may include interacting with a third party service.
Broadway looks like a perfect fit for the producer side but I didn’t see any complex example that includes multi step flows. Most examples do a local simple transformation and that’s it.
My current thought is to start a GenServer or GenStage on every new message that will handle the execution of the flow.
Most Liked
SirWerto
I would say that if you have in mind a workflow like this:
receive the request
save state to db
interact with a service
save state to db
interact with another service
save to db
...
you can just use a single process to do everything.
Using GenStage with multiple layers passing state between them would only complicate things.
And to define it, you can use a GenServer spawning more GenServers under a Dynamic Supervisor
joey_the_snake
One GenStage pipeline where the producers poll the table and then pass the information to the consumers who will do the processing. And then you can configure the concurrency for the producers/consumers to however much you need.
You might also be able to use a job processing library like Oban. I never used it myself but I believe the general idea is it stores jobs inside of a Postgres table and schedules them/runs them/updates their status.







