sherry

sherry

BeamTrail - a Postgres-backed durable process runner for BEAM apps

Hi,

I am building BeamTrail, a small BEAM library for durable business processes: retries, timers, signals, approval deadlines, operator controls, and crash recovery, backed by PostgreSQL.

It is not a Temporal clone and not a job queue. The goal is an embedded runtime for Erlang/Elixir/Phoenix apps that already run PostgreSQL and need long-running workflows without operating a separate workflow service.

The basic idea is:

  • Active runs are supervised OTP processes.
  • Business progress is stored as an append-only PostgreSQL event stream.
  • State is rebuilt by reducing events.
  • Snapshots and indexed projections are optimizations only.
  • Leases, fencing tokens, and expected sequence checks protect concurrent writers.
  • A scanner recovers unfinished runs whose lease is missing or expired.

The current use cases I am aiming at are things like:

  • order fulfillment with retryable external calls;
  • payment or webhook-driven sagas;
  • approval flows with deadlines;
  • long-running business processes that must survive VM restart;
  • AI agent/tool runs where the session memory is separate, but tool calls, approvals, timers, and recovery need durable
    execution boundaries.

Current implemented pieces:

  • PostgreSQL durable adapter and memory test adapter.
  • Append-only event log and reducer.
  • Expected-sequence append checks.
  • Per-run PostgreSQL append lock.
  • Lease renewal and fencing tokens.
  • Supervised per-run gen_statem active runner.
  • Scanner recovery with indexed recoverable-run projections.
  • Retries, timeouts, and crash-atomic failure decisions.
  • Cancel, park/resume, and manual requeue.
  • Version mismatch gates for in-flight attempt replay and decider workflows.
  • Durable signals and scanner-driven durable timers.
  • Executable approval-deadline pattern.
  • Crash recovery and PostgreSQL stress examples.
  • EUnit, PostgreSQL integration tests, xref, Dialyzer, and secret scan in CI.

Current limits:

  • No DAG, fan-out/fan-in, child workflows, or parallel command batches yet.
  • No first-class human task assignment UI.
  • No HTTP API or browser UI.
  • No SQL-native JSON payload inspection.
  • No built-in external side-effect deduplication.
  • No exactly-once execution of user callbacks.

The at-least-once boundary is explicit. BeamTrail provides stable idempotency keys, but workflow code must use them when calling external systems. If a VM dies after a callback performs an external side effect but before BeamTrail records the outcome, the same attempt can be re-entered with the same idempotency key.

How I currently think about the positioning:

  • If you only need background jobs, Oban is probably the better tool.
  • If you need a mature multi-language workflow platform, Temporal is probably the better tool.
  • BeamTrail is trying to sit in the smaller space between them: an embedded BEAM runtime for durable business processes, backed by PostgreSQL and shaped around OTP supervision.

What I would like feedback on:

  1. For Phoenix/Elixir apps, does this fill a real gap between Oban jobs and Temporal?
  2. Are the current APIs and semantics understandable from an Elixir user’s point of view, even though the library is written in Erlang?
  3. Would durable signals, timers, approval deadlines, and crash recovery be enough to try this in a small internal workflow?
  4. What would be the minimum missing feature before you would consider experimenting with it?
  5. Is PostgreSQL as the durable boundary acceptable, or would you expect a different storage/runtime shape?

Useful docs:

Repository:

There is also a GitHub Discussion for longer-form follow-up:

I am especially interested in practical feedback from people who run Phoenix/Elixir apps with background jobs, workflows, approvals, webhooks, or long-running business processes.

Thanks.

Most Liked

Yyeger

Yyeger

Hi,

Your lib seems very similar to mine which i posted a couple of days ago and started working on over a month ago. The main difference is that mine it’s in Elixir. We could work together maybe. Here is mine: Thread: Continuum - a library for durable and crash-resistant workflows Repo: GitHub - Yyeger/Continuum: OTP-native durable execution engine for Elixir. · GitHub

sherry

sherry

Hi, thanks for reaching out.

Yes, looks like we are exploring a very similar space from different angles. BeamTrail is more Erlang-first and event-sourced/state-machine oriented, while Continuum seems more Elixir workflow/replay oriented.

I’ll read through Continuum more carefully. Maybe there is room to share ideas or collaborate.

Where Next?

Popular in Announcing Top

jakub-zawislak
Hi everyone, I’m coming from the Symfony (PHP) framework. I like Phoenix, but it has a one thing that was build much better in the Symfo...
New
kelvinst
Hey everyone! Well, we made this lib a while ago and now we decided to finally go out and public with it! It’s a tool for creating and m...
New
treble37
Just looking for a little feedback on a tiny helper library I built - Sometimes I find the need to convert maps with atom keys to maps w...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 30486 241
New
devonestes
Introducing assertions, the library that helps you write really great test assertions! GitHub: GitHub - devonestes/assertions: Helpful a...
New
kip
Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir. I...
622 19610 194
New
trisolaran
Hi! :waving_hand: I would like to present LiveSelect, a little library that I wrote to easily add a dynamic selection input to your LV f...
198 11283 107
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 131117 1222
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement