Hey everyone, I have to build a workflow system like Zapier or n8n. Well, not as complete, but the same general idea. Essentially what I would like to achieve is to define a workflow declaratively. The workflow will have pre-existing nodes of any of these categories trigger node, one or more step nodes (that compute/modify state), and ends with an exit node(a stage that handles the store or sending the state).
Essentially model a set of steps as an acyclical graph utilizing genStage. I was wondering if genStage is the right tool for it or if it is an antipattern.
GenStage or Broadway is a good option for the runtime portion of an application like this. GenStage is great for those big “worker pool” in the sky with back-pressure and all that great stuff. I’d look at Broadway for an application like this and identify a queue/stream with guarantees for you (RabbitMQ/Kafka/SQS/GCP-PubSub/etc) as I expect using GenStage here will ultimately result in reimplementing some of Broadway’s features.
Instead I’d focus on your DAG model and evaluating the step dependency graph in a way that doesn’t couple you to a runtime system. Ideally you can have your workflow representation that can be evaluated with any different runtime system whether its in-memory with Task.async, Flow, Broadway, Broadway with different adapters, and so on. In fact I’d recommend dependency injecting this layer behind a Behaviour or Protocol of some kind i.e. Executor/Runner.
I made a proof of concept tool of a lazy DAG evaluation tool using libgraph that may be worth referencing:
It just handles stateless lamdba expressions with step dependencies and doesn’t do anything fancy like accumulations, rule inference, etc. So it’s not on hex - and is intended to be forked and customized for use.
What I’ve found works pretty well is a Dynamically Supervised GenServer that gets registered either with Elixir’s Registry or for a distributed cluster via something like Horde. The GenServer state holds the workflow/dag and consumes resulting events/facts of a given step to then identify and enqueue next runnable steps to the runner/execution layer (genstage, broadway, etc).
5 Likes
