sherry

sherry

BeamTrail - a Postgres-backed durable process runner for BEAM apps

Hi,

I am building BeamTrail, a small BEAM library for durable business processes: retries, timers, signals, approval deadlines, operator controls, and crash recovery, backed by PostgreSQL.

It is not a Temporal clone and not a job queue. The goal is an embedded runtime for Erlang/Elixir/Phoenix apps that already run PostgreSQL and need long-running workflows without operating a separate workflow service.

The basic idea is:

  • Active runs are supervised OTP processes.
  • Business progress is stored as an append-only PostgreSQL event stream.
  • State is rebuilt by reducing events.
  • Snapshots and indexed projections are optimizations only.
  • Leases, fencing tokens, and expected sequence checks protect concurrent writers.
  • A scanner recovers unfinished runs whose lease is missing or expired.

The current use cases I am aiming at are things like:

  • order fulfillment with retryable external calls;
  • payment or webhook-driven sagas;
  • approval flows with deadlines;
  • long-running business processes that must survive VM restart;
  • AI agent/tool runs where the session memory is separate, but tool calls, approvals, timers, and recovery need durable
    execution boundaries.

Current implemented pieces:

  • PostgreSQL durable adapter and memory test adapter.
  • Append-only event log and reducer.
  • Expected-sequence append checks.
  • Per-run PostgreSQL append lock.
  • Lease renewal and fencing tokens.
  • Supervised per-run gen_statem active runner.
  • Scanner recovery with indexed recoverable-run projections.
  • Retries, timeouts, and crash-atomic failure decisions.
  • Cancel, park/resume, and manual requeue.
  • Version mismatch gates for in-flight attempt replay and decider workflows.
  • Durable signals and scanner-driven durable timers.
  • Executable approval-deadline pattern.
  • Crash recovery and PostgreSQL stress examples.
  • EUnit, PostgreSQL integration tests, xref, Dialyzer, and secret scan in CI.

Current limits:

  • No DAG, fan-out/fan-in, child workflows, or parallel command batches yet.
  • No first-class human task assignment UI.
  • No HTTP API or browser UI.
  • No SQL-native JSON payload inspection.
  • No built-in external side-effect deduplication.
  • No exactly-once execution of user callbacks.

The at-least-once boundary is explicit. BeamTrail provides stable idempotency keys, but workflow code must use them when calling external systems. If a VM dies after a callback performs an external side effect but before BeamTrail records the outcome, the same attempt can be re-entered with the same idempotency key.

How I currently think about the positioning:

  • If you only need background jobs, Oban is probably the better tool.
  • If you need a mature multi-language workflow platform, Temporal is probably the better tool.
  • BeamTrail is trying to sit in the smaller space between them: an embedded BEAM runtime for durable business processes, backed by PostgreSQL and shaped around OTP supervision.

What I would like feedback on:

  1. For Phoenix/Elixir apps, does this fill a real gap between Oban jobs and Temporal?
  2. Are the current APIs and semantics understandable from an Elixir user’s point of view, even though the library is written in Erlang?
  3. Would durable signals, timers, approval deadlines, and crash recovery be enough to try this in a small internal workflow?
  4. What would be the minimum missing feature before you would consider experimenting with it?
  5. Is PostgreSQL as the durable boundary acceptable, or would you expect a different storage/runtime shape?

Useful docs:

Repository:

There is also a GitHub Discussion for longer-form follow-up:

I am especially interested in practical feedback from people who run Phoenix/Elixir apps with background jobs, workflows, approvals, webhooks, or long-running business processes.

Thanks.

Most Liked

Yyeger

Yyeger

Hi,

Your lib seems very similar to mine which i posted a couple of days ago and started working on over a month ago. The main difference is that mine it’s in Elixir. We could work together maybe. Here is mine: Thread: Continuum - a library for durable and crash-resistant workflows Repo: GitHub - Yyeger/Continuum: OTP-native durable execution engine for Elixir. · GitHub

sherry

sherry

Hi, thanks for reaching out.

Yes, looks like we are exploring a very similar space from different angles. BeamTrail is more Erlang-first and event-sourced/state-machine oriented, while Continuum seems more Elixir workflow/replay oriented.

I’ll read through Continuum more carefully. Maybe there is room to share ideas or collaborate.

Where Next?

Popular in Announcing Top

Crowdhailer
The latest release of Ace (0.10.0) includes serving content over HTTP/2. I have started writing a webserver to teach my self more about...
New
josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
mikehostetler
I’m excited to announce Jido, a framework providing foundational primitives for building autonomous agent systems in Elixir. While develo...
New
Eiji
ExApi is a library that I’m developing now and hope release soon This library will allow to: list all apis list all api implementation...
New
Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
achempion
Hi, I would like to tell about my initiative to further maintain and develop Waffle project which is the fork of Arc library. The progre...
New
michalmuskala
Hello everybody. I have just released Jason - a new JSON library. You might be wondering, why do we need a new library? The primary foc...
New
Qqwy
TypeCheck: Fast and flexible runtime type-checking for your Elixir projects. Core ideas Type- and function specifications are const...
336 14482 100
New
tfwright
After working on it for a couple of months and using it in production for most of that time, today I’ve released LiveAdmin, a LiveView ba...
New
New

Other popular topics Top

siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 43487 311
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52673 488
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement