Any tips on building a workflow engine similar to n8n?

Useful project.

Suggestions/inquiries:

Can you create examples/README.md and/or ref it in the main README.md?

Are there external integrations, either with mocks or disabled by default when running mix test with clear tags?

Are you doing static analysis with dialyzer?

Perhaps consider adding a runner for github to show the green check, indicating all default tests pass?

Thanks @gtcode, external integrations are implemented at main application.
I’m working to test integrating to our main application. Keep back and ford to refactor to adapt to application needs. Lots of unknown here.

Glific is a graph-based chatbot creation platform. The backend is in Elixir which handles the floweditor engine, like executing a graph of nodes. Maybe you can check this out.
https://github.com/glific/glific

Thank you, I’m looking into it

I went through the codebase, everything I would expect of a workflow engine is there. Very nice implementation, good work.

What I would expect from an engine is the ability to recover from:

• kill -9 erlang vm
• rm -rf /local/data
• k8s cluster delete

When i start my elixir cluster again, absolutely nothing should be lost and the system should eventually return to my 10000 workflows to a running state as if nothing happened.

That is one of the hard parts about these things.

There a few open source examples for your examination:
https://github.com/cadence-workflow/cadence
temporal.io · GitHub (more programmer focused)
Oban Pro has the ability to do run workflows, however it’s closed source (possibly wrong here)

^ those are like end game solutions, it’s not actually too hard to track workflow execution state using postgres or similar to get durability. The simple solution works just as well at a normal scale

I’d also look at business process engines, same idea, but just different style of implementation.

Awesome work though Looking forward to seeing how it progresses over time.

While reading your post, postgres came to mind for durability, and then you mentioned it!

I’m working on related problems. Thanks to reading the posts, I just thought of these integration concerns related to durability and resiliency:

Step Recovery

Pure Deterministic (No Side Effects): Mathematical calculations, data transformations, pure functions. These are truly idempotent - same input always produces same output with no external impact. Retry freely without concern. Examples: JSON parsing, mathematical computations, string manipulations.

Deterministic with Side Effects: Database writes, file operations, REST API calls with predictable behavior. The logic is deterministic but external state matters. Implement conditional idempotency using techniques like: check-then-act patterns, unique transaction IDs, upserts over inserts, and proper state validation before retry. Examples: user registration, inventory updates, email notifications with deduplication keys.

Non-Deterministic without Side Effects: AI inference, random number generation, heuristic algorithms that don’t modify external state. Use semantic idempotency - ensure the intent/goal remains consistent even if exact outputs differ. Checkpoint based on meaningful progress rather than exact state. Examples: content generation, recommendation algorithms, data analysis with ML models.

Non-Deterministic with Side Effects: AI agents making API calls, LLM-powered workflows that interact with external systems, adaptive processes that learn from environment. The most complex category - combine semantic idempotency with careful side effect management. Implement context-aware checkpointing that captures both progress and environmental assumptions, validate external state on recovery, and design for graceful adaptation when conditions change. Examples: AI agents conducting research and filing reports, automated trading systems, dynamic workflow orchestration.

Workflow Management

Design Strategy: Structure your workflow engine with step classification at the core - each step declares its category (pure deterministic, deterministic+side effects, non-deterministic, non-deterministic+side effects) which drives the recovery logic. Store step definitions, execution state, and checkpoints in Postgres with proper transaction boundaries.

Implementation Approach: Use Postgres transactions to atomically update step status and checkpoint data. For deterministic steps, store minimal state (input/output hashes, completion flags). For non-deterministic steps, serialize rich checkpoint data including context, assumptions, and partial progress. Implement a recovery coordinator that reads the step type and applies the appropriate strategy: simple retry for pure functions, conditional retry with state validation for side-effect deterministic steps, checkpoint-based resumption for non-deterministic work. Leverage Postgres’s ACID properties to ensure your workflow state itself remains consistent even when individual steps fail, and use row-level locking to handle concurrent workflow executions safely.​​​​​​​​​​​​​​​​

Please share revisions/correction/enhancements based on your knowledge and experience.

Thank @nerdyworm for your references.

Recovering workflow execution is what I’m stuck with. Thanks @gtcode for pointing out above, it’s very clear. your comments lighted up my mind. I’m currently overloaded by too many problems, so I decide to integrate with our main application first to test a real workflow. Then I’ll come back to add more improvement later. I know that will break lots of things and take much effort to refactor. But I think it’s better than get stuck of trying solving complicated problems and make no real progress

I’ll update progress later when I come back to play with these recovery strategies.

Hi, we have an internal workflow engine that quite different from n8n (code-only). Two thoughts. The API may be easier to read if you take inspiration from Oban workflows: Oban.Pro.Workflow — Oban Pro v1.6.2. (Btw we are Oban Pro users and highly recommend!)

    Workflow.new()
    |> Workflow.add(:a, new_echo(1))
    |> Workflow.add(:b, new_echo(2), deps: :a)
    |> Workflow.add(:c, new_echo(3), deps: :b)
    |> Workflow.add(:d, new_echo(4), deps: :b)
    |> Workflow.add(:e, new_echo(5), deps: [:c, :d])
    |> Oban.insert_all()

If you use a library like GitHub - bitwalker/libgraph: A graph data structure library for Elixir projects · GitHub validation/orchestration might be easier.

On another note, if you are interested in durable execution, we took a lot of inspiration from build systems and Nix. A lot of the same problems have been solved by compilers.

For example, imagine you need to compile a project and there is a compile error. What happens? A subset of the artifacts get stored and cached. Then you fix the error, recompile, and it picks up where you left off. Some of the compilation work is skipped over (no-op) and the ones for which you fixed the error are able to keep going.

Hope that helps, happy to discuss further ideas. Having something like n8n for Elixir would be amazing!

I would look at (or use) Oban and Ash Reactor. Maybe someone in the Ash project would be interested in collaborating.

IMO gui workflow definition is optional. Kestra uses YAML files…

N8N has security/privacy/sustainability problems. An open-source Elixir-based workflow engine would be great.

We built something like this at Bridge Connector, before the company went under (bad mgmt).

Let me know what you’re struggling on at the moment and maybe I can help.

Take a look at OpenFn if you’re curious to see other approaches too

Questions:

• presumably your user is someone building a workflow from a UI from individual steps?
• is there branching logic?
• dependency concepts? (e.g. branches combining)
• are the integrations using your API keys or theirs? is noisy neighbor/workflow a concern?
• is it json schema for the workflow inputs?

Let me know, and I’d be happy to hop on a call sometime too