stjefim
Durable Workflow Execution (temporal alternative for elixir)
Hello!
Suppose you are building workflow (order / task / payment) processing system with the following requirements:
- Each workflow consists of several steps.
- Each step can fail (throw exception or return error) or timeout and need to be retried according to retry policy.
- State machine - choice of next step in a workflow depends on the result of the previous step.
- Durable execution - workflows and steps taken and their success or failure are persisted to DB, so that workflows are never (e.g. if server fails) lost and can be resumed from the next step.
- State visibility - Web UI to monitor workflow progress.
- Other less important requirements, e.g. auto scaling of workers executing steps, cancellation of workflows, signalling to workflows, …
I believe this system is very often required for many websites and is a good fit to Elixir, however, I believe there is no ready to use solution in Elixir and developing such a system for each website separately is a waste of time and resources. Do you agree with this statement? How would you go about developing such a system in Elixir? Writing from ground up? May be some framework I have missed?
Solutions I have considered:
- Temporal - almost ideal fit to the system requirements, however, no SDK for Elixir (unofficial SDK in development, should be ready within 2-3 months) and I believe is not an ideal fit to Elixir. I think a better solution for Elixir would be to run Service managing state and Workers executing steps together in BEAM. This would allow to e.g., leverage BEAM’s message passing, use cache inside BEAM (e.g. Cachex) and in general have less dependencies.
- Oban - provides durable execution, retries, timeouts, however, not a state machine as is made for durable background job (not workflow) processing. While at first it seems that adding state machine is not a problem at all, I believe this would require many tricks and hacks, thus, poor code clarity and poor state visibility - Oban’s Web UI is not made for this use case. E.g., I have considered using Oban Workflows with ignore_discarded, however, this required scheduling steps for both cases (previous step failed or succeeded) and creates messy code as well as poor workflow visibility.
Most Liked
webofbits
I’ve been poking at this same problem space in Squid Mesh:
It’s still early, but the shape I’m experimenting with is durable workflow runs inside an existing Phoenix/OTP app: persisted runs/steps/attempts/audit events, retries, waits, approvals, replay, cancellation, and inspection.
The boundary is somewhere between a job queue and a separate Temporal-style service. The host app still owns the repo, deployment, and queue/executor; Squid Mesh owns the workflow state and recovery/inspection bits.
Lately I’ve been moving more of the runtime toward Jido primitives: actions for step execution, agents for rebuildable workflow/dispatch coordination, and Jido journals as the durable fact log. The goal is for the runtime to be replayable from durable facts instead of treating worker/job state as the source of truth.
Also, I’ve also been experimenting with a BedrockDB-backed execution path for leases, redelivery, and recovery semantics, while keeping the workflow layer embedded in the app.
There’s also a small read-only LiveView dashboard for it:
Not production-ready, and I’m still figuring out some of the runtime boundaries, but this thread is very close to the tradeoffs I’ve been thinking about. I’d be curious if this shape matches what others have wanted from an Elixir-native workflow layer, or if I’m drawing the boundary in the wrong place.
MrDoops
The sweet spot in most use cases is just doing a straight workflow → graph serialization with a viz tool like Mermaid / Cytoscape / DOT e.g. Runic.Workflow — Runic v0.1.0-alpha.8
Usually just throwing this in Livebook or mermaid.live is enough to debug or document the workflow. I’ve found a read-only view in an admin panel something like this to just render the graph on a show page that shows execution history, errors, traces, etc tends to solve most problems with the least effort.
I’ve also done more on drag and drop builders and have it working pretty well on top of Liveview and Reactflow but its still specific to that app moreso than a library that I can open source. It ends up being a full app because you kind of want a component library which means you have to store components, and some components need parameters or forms to materialize so you need to persist forms in the database and do data driven validation. Then some components are only compatible with other components so there’s dataflow contracts to worry about. Then undo/redo, and some view/model separation where certain components should be rendered on the canvas different than the underlying dataflow its compiled to, etc, etc…
Frankly its a ton of work and I wonder how valuable it is from a UI ROI perspective considering how much work it is to get something usable enough to be worthwhile over editing code or having an agent write your workflow (which I’ve been doing more and more with Runic).
There’s some new Elixir libraries like GitHub - thanos/ExFlowGraph: LiveView-native node editor for Elixir · GitHub and GitHub - rocket4ce/live_flow · GitHub but I haven’t tried them yet.
I’ll probably release some tools for this with Runic eventually but haven’t been happy with the ROI or how specific the UX decisions are to the application they’re for.
I’ve actually released Runic on Hex recently (currently in alpha): it has a scheduling and execution layer with durable execution and some neat event-sourced based check-pointing and re-hydration capabilities for long running workflows so its worth checking out: runic | Hex .
MrDoops
I’m building something like this on top of Runic. No guarantees on when the web UI & durable execution is at a production ready state though. I have a durable runner implemented but still working on check-pointing for long running workflows. There’s also quite a bit one would normally want in this sort of thing like triggers (e.g. webhooks, messaging system integrations), CRON / time scheduled workflows, and so on.
I’d recommend Oban if you want a durable execution with graph based workflows today.
While it’s not a DAG: GitHub - commanded/commanded: Use Commanded to build Elixir CQRS/ES applications · GitHub is also this sort of thing but with DDD/CQRS abstractions.
Paulo released handoff recently which uses dags: GitHub - polvalente/handoff: Distributed graph execution in Elixir · GitHub
Last Post!
yoavgeva
This is very close to the direction I also ended up taking with FerricStore / FerricFlow.
The boundary I wanted is also somewhere between a job queue and a Temporal-style service, but with a bit
different tradeoff.
I dont want the runtime to own all my application code like Temporal. I also dont want to keep using
queues and then build workflow state around them every time.
What I wanted was a high throughput workflow engine where:
- my app still owns the business code / workers
- FerricFlow owns the durable workflow state
- leases, retries, attempts, history, signals and recovery are first class
- the workflow can be inspected and repaired
- the same durable store can also do KV / Redis-style data structures
So the model is more explicit state machine than durable code replay.
Something like:
FLOW.CREATE order-1 STATE charge
worker claims charge
worker does the side effect
FLOW.TRANSITION order-1 shipped
FerricStore keeps the durable state, lease/fencing token, retry info, history, values, signals, etc.
The thing you mention around persisted runs/steps/attempts/audit events, waits, approvals, cancellation
and inspection is exactly the pain I am trying to solve too. FerricFlow also added governance concepts
now: budgets, approvals, external effect tracking, distributed limits and circuit breakers.
I already use the HA cluster in my own projects, and in my company we started using it for small stuff
also. Feedback is very positive until now, but I still see it as beta and not finished 1.0.
Current repo:
GitHub - ferricstore/ferricstore: Durable KV and workflow state engine · GitHub
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"