Durable Workflow Execution (temporal alternative for elixir)

This is something we plan to fix. A dedicated workflow view is slated for Oban Web.

There’s no need to schedule separate jobs for the result of the previous job. With recorded jobs it’s simple to fetch the result from the previous step, or even to check the status of the previous step if needed. Here’s an example:

@impl Oban.Pro.Worker
def process(job) do
  case Workflow.get_recorded(job, :previous_step) do
    nil ->
      # It failed and nothing was recorded, handle that case.
    value ->
      # It worked, carry on
  end
end

I agree, another option is to schedule single job which handles both cases, however, in that case at the next step (3rd one) job would need to handle 4 cases (1st failed 2nd succeeded; 1st failed 2nd failed, 1st succeeded 2nd succeeded; 1st succeeded 2nd failed). On the next step 8 cases. To sum up, if state transition graph is at least a bit complex, this solution creates messy code. As well as it decreases visibility as now single job could represent completely different operations (e.g. cleanup after previous job assuming it failed, or processing of the next step assuming previous job succeeded), thus, from the look at the single job (without looking at all previous this workflow jobs) in the Web UI it is unclear what is happening.

I’ve been thinking about this a bit too, having just discovered Temporal recently. I have use-cases where I think I need python workers (not just an embedded script a la Pythonx) due to library support and it seems it would be pretty nice to use the same orchestration layer - no idea what it actually feels like in practice, of course. I suppose it provides Elixir with another path for integration within heterogeneous environments.

I’m building something like this on top of Runic. No guarantees on when the web UI & durable execution is at a production ready state though. I have a durable runner implemented but still working on check-pointing for long running workflows. There’s also quite a bit one would normally want in this sort of thing like triggers (e.g. webhooks, messaging system integrations), CRON / time scheduled workflows, and so on.

I’d recommend Oban if you want a durable execution with graph based workflows today.

While it’s not a DAG: GitHub - commanded/commanded: Use Commanded to build Elixir CQRS/ES applications · GitHub is also this sort of thing but with DDD/CQRS abstractions.

Paulo released handoff recently which uses dags: GitHub - polvalente/handoff: Distributed graph execution in Elixir · GitHub

Screenshot from 2025-05-24 09-51-251200×816 38.5 KB

That’s a good point about CQRS and Commanded - quite a robust native Elixir option. I hadn’t even heard of handoff - very cool looking!

Hey, I wonder if you ever returned to your idea to build a visual layer on top of Runic?

The sweet spot in most use cases is just doing a straight workflow → graph serialization with a viz tool like Mermaid / Cytoscape / DOT e.g. Runic.Workflow — Runic v0.1.0-alpha.8

Usually just throwing this in Livebook or mermaid.live is enough to debug or document the workflow. I’ve found a read-only view in an admin panel something like this to just render the graph on a show page that shows execution history, errors, traces, etc tends to solve most problems with the least effort.

I’ve also done more on drag and drop builders and have it working pretty well on top of Liveview and Reactflow but its still specific to that app moreso than a library that I can open source. It ends up being a full app because you kind of want a component library which means you have to store components, and some components need parameters or forms to materialize so you need to persist forms in the database and do data driven validation. Then some components are only compatible with other components so there’s dataflow contracts to worry about. Then undo/redo, and some view/model separation where certain components should be rendered on the canvas different than the underlying dataflow its compiled to, etc, etc…

Frankly its a ton of work and I wonder how valuable it is from a UI ROI perspective considering how much work it is to get something usable enough to be worthwhile over editing code or having an agent write your workflow (which I’ve been doing more and more with Runic).

There’s some new Elixir libraries like GitHub - thanos/ExFlowGraph: LiveView-native node editor for Elixir · GitHub and GitHub - rocket4ce/live_flow · GitHub but I haven’t tried them yet.

I’ll probably release some tools for this with Runic eventually but haven’t been happy with the ROI or how specific the UX decisions are to the application they’re for.

I’ve actually released Runic on Hex recently (currently in alpha): it has a scheduling and execution layer with durable execution and some neat event-sourced based check-pointing and re-hydration capabilities for long running workflows so its worth checking out: runic | Hex .

I’ve been poking at this same problem space in Squid Mesh:

It’s still early, but the shape I’m experimenting with is durable workflow runs inside an existing Phoenix/OTP app: persisted runs/steps/attempts/audit events, retries, waits, approvals, replay, cancellation, and inspection.

The boundary is somewhere between a job queue and a separate Temporal-style service. The host app still owns the repo, deployment, and queue/executor; Squid Mesh owns the workflow state and recovery/inspection bits.

Lately I’ve been moving more of the runtime toward Jido primitives: actions for step execution, agents for rebuildable workflow/dispatch coordination, and Jido journals as the durable fact log. The goal is for the runtime to be replayable from durable facts instead of treating worker/job state as the source of truth.

Also, I’ve also been experimenting with a BedrockDB-backed execution path for leases, redelivery, and recovery semantics, while keeping the workflow layer embedded in the app.

There’s also a small read-only LiveView dashboard for it:

Not production-ready, and I’m still figuring out some of the runtime boundaries, but this thread is very close to the tradeoffs I’ve been thinking about. I’d be curious if this shape matches what others have wanted from an Elixir-native workflow layer, or if I’m drawing the boundary in the wrong place.

This is very close to the direction I also ended up taking with FerricStore / FerricFlow.

The boundary I wanted is also somewhere between a job queue and a Temporal-style service, but with a bit
different tradeoff.

I dont want the runtime to own all my application code like Temporal. I also dont want to keep using
queues and then build workflow state around them every time.

What I wanted was a high throughput workflow engine where:

• my app still owns the business code / workers
• FerricFlow owns the durable workflow state
• leases, retries, attempts, history, signals and recovery are first class
• the workflow can be inspected and repaired
• the same durable store can also do KV / Redis-style data structures

So the model is more explicit state machine than durable code replay.

Something like:

  FLOW.CREATE order-1 STATE charge
  worker claims charge
  worker does the side effect
  FLOW.TRANSITION order-1 shipped

FerricStore keeps the durable state, lease/fencing token, retry info, history, values, signals, etc.

The thing you mention around persisted runs/steps/attempts/audit events, waits, approvals, cancellation
and inspection is exactly the pain I am trying to solve too. FerricFlow also added governance concepts
now: budgets, approvals, external effect tracking, distributed limits and circuit breakers.

I already use the HA cluster in my own projects, and in my company we started using it for small stuff
also. Feedback is very positive until now, but I still see it as beta and not finished 1.0.

Current repo:

GitHub - ferricstore/ferricstore: Durable KV and workflow state engine · GitHub

I’ve been working a lot on GitHub - Yyeger/Continuum: OTP-native durable execution engine for Elixir. · GitHub . i would say it covers something similar, it’s also avaiable on Hex continuum | Hex .