sherry

sherry

BeamTrail - a Postgres-backed durable process runner for BEAM apps

Hi,

I am building BeamTrail, a small BEAM library for durable business processes: retries, timers, signals, approval deadlines, operator controls, and crash recovery, backed by PostgreSQL.

It is not a Temporal clone and not a job queue. The goal is an embedded runtime for Erlang/Elixir/Phoenix apps that already run PostgreSQL and need long-running workflows without operating a separate workflow service.

The basic idea is:

  • Active runs are supervised OTP processes.
  • Business progress is stored as an append-only PostgreSQL event stream.
  • State is rebuilt by reducing events.
  • Snapshots and indexed projections are optimizations only.
  • Leases, fencing tokens, and expected sequence checks protect concurrent writers.
  • A scanner recovers unfinished runs whose lease is missing or expired.

The current use cases I am aiming at are things like:

  • order fulfillment with retryable external calls;
  • payment or webhook-driven sagas;
  • approval flows with deadlines;
  • long-running business processes that must survive VM restart;
  • AI agent/tool runs where the session memory is separate, but tool calls, approvals, timers, and recovery need durable
    execution boundaries.

Current implemented pieces:

  • PostgreSQL durable adapter and memory test adapter.
  • Append-only event log and reducer.
  • Expected-sequence append checks.
  • Per-run PostgreSQL append lock.
  • Lease renewal and fencing tokens.
  • Supervised per-run gen_statem active runner.
  • Scanner recovery with indexed recoverable-run projections.
  • Retries, timeouts, and crash-atomic failure decisions.
  • Cancel, park/resume, and manual requeue.
  • Version mismatch gates for in-flight attempt replay and decider workflows.
  • Durable signals and scanner-driven durable timers.
  • Executable approval-deadline pattern.
  • Crash recovery and PostgreSQL stress examples.
  • EUnit, PostgreSQL integration tests, xref, Dialyzer, and secret scan in CI.

Current limits:

  • No DAG, fan-out/fan-in, child workflows, or parallel command batches yet.
  • No first-class human task assignment UI.
  • No HTTP API or browser UI.
  • No SQL-native JSON payload inspection.
  • No built-in external side-effect deduplication.
  • No exactly-once execution of user callbacks.

The at-least-once boundary is explicit. BeamTrail provides stable idempotency keys, but workflow code must use them when calling external systems. If a VM dies after a callback performs an external side effect but before BeamTrail records the outcome, the same attempt can be re-entered with the same idempotency key.

How I currently think about the positioning:

  • If you only need background jobs, Oban is probably the better tool.
  • If you need a mature multi-language workflow platform, Temporal is probably the better tool.
  • BeamTrail is trying to sit in the smaller space between them: an embedded BEAM runtime for durable business processes, backed by PostgreSQL and shaped around OTP supervision.

What I would like feedback on:

  1. For Phoenix/Elixir apps, does this fill a real gap between Oban jobs and Temporal?
  2. Are the current APIs and semantics understandable from an Elixir user’s point of view, even though the library is written in Erlang?
  3. Would durable signals, timers, approval deadlines, and crash recovery be enough to try this in a small internal workflow?
  4. What would be the minimum missing feature before you would consider experimenting with it?
  5. Is PostgreSQL as the durable boundary acceptable, or would you expect a different storage/runtime shape?

Useful docs:

Repository:

There is also a GitHub Discussion for longer-form follow-up:

I am especially interested in practical feedback from people who run Phoenix/Elixir apps with background jobs, workflows, approvals, webhooks, or long-running business processes.

Thanks.

Most Liked

Yyeger

Yyeger

Hi,

Your lib seems very similar to mine which i posted a couple of days ago and started working on over a month ago. The main difference is that mine it’s in Elixir. We could work together maybe. Here is mine: Thread: Continuum - a library for durable and crash-resistant workflows Repo: GitHub - Yyeger/Continuum: OTP-native durable execution engine for Elixir. · GitHub

sherry

sherry

Hi, thanks for reaching out.

Yes, looks like we are exploring a very similar space from different angles. BeamTrail is more Erlang-first and event-sourced/state-machine oriented, while Continuum seems more Elixir workflow/replay oriented.

I’ll read through Continuum more carefully. Maybe there is room to share ideas or collaborate.

Where Next?

Popular in Announcing Top

gabrielpoca
Hello everyone! I want to share with you something that I’m really proud of: https://stillstatic.io/ Still is a static site builder for...
New
oltarasenko
Dear Elixir community, After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.1...
New
cjen07
parameterized pipe in elixir: |n> edit: negative index in |n> and mixed usage with |> are supported example: use ParamPipe ...
New
versilov
Could not wait for the missing Elixir ML libraries to appear, so, I wrote one myself, taking https://github.com/sdwolfz/exlearn as a foun...
New
hpopp
After just over two years in development, this latest version of Pigeon is what I finally consider done in regards to my original vision ...
New
Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Thank...
New
markmark206
simple_feature_flags is a tiny package that lets you turn features on or off based on which environment (e.g. localhost, staging, product...
New
OvermindDL1
Been making an MLElixir thing (not released yet…) for fun in spare time in the past day. I’m just trying to see how much I can get an ML...
132 13966 106
New
marcuslankenau
I feel kind of stuck with the absence of a proper xml library for Elixir. Currently I use SweetXML which was ok for me more or less to pa...
New
pkrawat1
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement