sherry

sherry

BeamTrail - a Postgres-backed durable process runner for BEAM apps

Hi,

I am building BeamTrail, a small BEAM library for durable business processes: retries, timers, signals, approval deadlines, operator controls, and crash recovery, backed by PostgreSQL.

It is not a Temporal clone and not a job queue. The goal is an embedded runtime for Erlang/Elixir/Phoenix apps that already run PostgreSQL and need long-running workflows without operating a separate workflow service.

The basic idea is:

  • Active runs are supervised OTP processes.
  • Business progress is stored as an append-only PostgreSQL event stream.
  • State is rebuilt by reducing events.
  • Snapshots and indexed projections are optimizations only.
  • Leases, fencing tokens, and expected sequence checks protect concurrent writers.
  • A scanner recovers unfinished runs whose lease is missing or expired.

The current use cases I am aiming at are things like:

  • order fulfillment with retryable external calls;
  • payment or webhook-driven sagas;
  • approval flows with deadlines;
  • long-running business processes that must survive VM restart;
  • AI agent/tool runs where the session memory is separate, but tool calls, approvals, timers, and recovery need durable
    execution boundaries.

Current implemented pieces:

  • PostgreSQL durable adapter and memory test adapter.
  • Append-only event log and reducer.
  • Expected-sequence append checks.
  • Per-run PostgreSQL append lock.
  • Lease renewal and fencing tokens.
  • Supervised per-run gen_statem active runner.
  • Scanner recovery with indexed recoverable-run projections.
  • Retries, timeouts, and crash-atomic failure decisions.
  • Cancel, park/resume, and manual requeue.
  • Version mismatch gates for in-flight attempt replay and decider workflows.
  • Durable signals and scanner-driven durable timers.
  • Executable approval-deadline pattern.
  • Crash recovery and PostgreSQL stress examples.
  • EUnit, PostgreSQL integration tests, xref, Dialyzer, and secret scan in CI.

Current limits:

  • No DAG, fan-out/fan-in, child workflows, or parallel command batches yet.
  • No first-class human task assignment UI.
  • No HTTP API or browser UI.
  • No SQL-native JSON payload inspection.
  • No built-in external side-effect deduplication.
  • No exactly-once execution of user callbacks.

The at-least-once boundary is explicit. BeamTrail provides stable idempotency keys, but workflow code must use them when calling external systems. If a VM dies after a callback performs an external side effect but before BeamTrail records the outcome, the same attempt can be re-entered with the same idempotency key.

How I currently think about the positioning:

  • If you only need background jobs, Oban is probably the better tool.
  • If you need a mature multi-language workflow platform, Temporal is probably the better tool.
  • BeamTrail is trying to sit in the smaller space between them: an embedded BEAM runtime for durable business processes, backed by PostgreSQL and shaped around OTP supervision.

What I would like feedback on:

  1. For Phoenix/Elixir apps, does this fill a real gap between Oban jobs and Temporal?
  2. Are the current APIs and semantics understandable from an Elixir user’s point of view, even though the library is written in Erlang?
  3. Would durable signals, timers, approval deadlines, and crash recovery be enough to try this in a small internal workflow?
  4. What would be the minimum missing feature before you would consider experimenting with it?
  5. Is PostgreSQL as the durable boundary acceptable, or would you expect a different storage/runtime shape?

Useful docs:

Repository:

There is also a GitHub Discussion for longer-form follow-up:

I am especially interested in practical feedback from people who run Phoenix/Elixir apps with background jobs, workflows, approvals, webhooks, or long-running business processes.

Thanks.

Most Liked

Yyeger

Yyeger

Hi,

Your lib seems very similar to mine which i posted a couple of days ago and started working on over a month ago. The main difference is that mine it’s in Elixir. We could work together maybe. Here is mine: Thread: Continuum - a library for durable and crash-resistant workflows Repo: GitHub - Yyeger/Continuum: OTP-native durable execution engine for Elixir. · GitHub

sherry

sherry

Hi, thanks for reaching out.

Yes, looks like we are exploring a very similar space from different angles. BeamTrail is more Erlang-first and event-sourced/state-machine oriented, while Continuum seems more Elixir workflow/replay oriented.

I’ll read through Continuum more carefully. Maybe there is room to share ideas or collaborate.

Where Next?

Popular in Announcing Top

sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
martinthenth
Hello everybody :wave: Recently, some of my colleagues talked about database ids and uuids and their problems, and I remembered the pain...
New
ityonemo
Currently just starting out on a new mini-project - getting zig NIFs to run in elixir. https://github.com/ityonemo/zigler The idea here...
New
ostinelli
Let’s write a database! Well not really, but I think it’s a little sad that there doesn’t seem to be a simple in-memory distributed KV da...
New
brainlid
LangChain is short for Language Chain. An LLM, or Large Language Model, is the “Language” part. This library makes it easier for Elixir a...
New
cjen07
parameterized pipe in elixir: |n> edit: negative index in |n> and mixed usage with |> are supported example: use ParamPipe ...
New
achempion
Hi, I would like to tell about my initiative to further maintain and develop Waffle project which is the fork of Arc library. The progre...
New
OvermindDL1
Been making an MLElixir thing (not released yet…) for fun in spare time in the past day. I’m just trying to see how much I can get an ML...
132 13966 106
New
anshuman23
Hello all, I have been working on my proposed project called Tensorflex as part of Google Summer of Code 2018.. Tensorflex can be used f...
New
pkrawat1
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
New

Other popular topics Top

gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New

We're in Beta

About us Mission Statement