Building an AI agent runtime moving from TypeScript to Elixir/OTP - can you help/compare notes?

durvia

Anyone running long-lived stateful processes on BEAM? We’re building an AI agent runtime and would love to compare notes.

We’re a small team (5 devs from China and Italy) building AI agent runtime in TypeScript. It’s on Bun (~60K lines src, ~40K tests, Apache 2.0, open-sourcing soon) that has been running on for ~3 months, processing ~100B tokens and ~300K agent actions. The agents running inside actively contribute to bootstrap development (under human review and refactoring)

What we have done: It ingests events from external cron/report/feed/webhook and platform messages into a unified envelope, routes them to exact agent under its address. Agents run serially within a node, concurrently across addresses, each with independent orchestration (toolsets, cwd, models, prompts, sub-agent teams). Nodes form dynamic topology via communication edges, that means agents can address each other to deliver messages, works, or directly check context on other nodes. All agent actions have durable trajectories while streamed to Web/IM/TUI/OAI Endpoint where users can inspect and intervene any agent, all within one JS process.

How I ended up here

Our architecture works fine for a single instance with ~100 concurrent agents. But I am looking for something genuinely new: high-concurrency agent clusters serving factory and enterprise workloads. The current open-source agent harness ecosystem has not reached there.

We manually built lifecycle management for these agents in TypeScript, and while exploring multi-agent fault isolation, live capability changes, and address-based routing. Recently, I accidentally discovered that OTP had already solved what troubles me decades ago:

Every node needs an address that serves as routing key, supervision path, and storage partition key, so anything with an API becomes a node in the agent mesh. This is what {:via, Registry, key} already gives in BEAM.
Agents are organized by root → platform → … → channel → worker. We wrote error isolation and recovery logic by hand. When I read the OTP supervisor documentation I recognized we had built a worse supervisor tree.
We redesigned agent sessions as a logical window over an append-only stream per address. This is essentially process state reconstruction after restart, what init/1 plus message replay gives BEAM processes.

What I’ve tried

I compared four candidates (Elixir, C#/Orleans, Java/Pekko, Rust/ractor) with hands-on experiments. Elixir came out ahead on supervision, hot code replacement, per-process GC, and the set-theoretic type system fitting our data shapes. The gaps I see are no user-defined type signatures yet and a smaller ecosystem for some dependencies.

I also ran some Elixir experiments, like:

Macro-level pattern matching: independent implementations (Claude/GLM/GPT) of or-patterns with compile-time exhaustiveness checking, exploring how well AI models write Elixir and how the macro system goes.
Self-diagnosing agent: A tool-calling loop that inspects a running GenServer using :sys.get_state and :dbg tracing, diagnoses a seeded bug from behavioral traces alone, then hot-patches via Code.compile_string + :code.load_binary without restarting the process or reading the source code. Session transcript attached below.

My Questions

I’d love to hear from anyone who has experience with long-lived stateful processes on BEAM:

Have you built systems where processes live for days/weeks with accumulated state?
How do you handle hot code changes in production for stateful GenServers?
Has anyone tried building AI/LLM agent systems on BEAM? I found bash interpretation in Elixir, curious if there are others.

Also looking for collaborators

None of us have production Elixir experience. We have TS codes and clear architectural direction, but need someone who has shipped OTP in production to tell where our instincts are right and where we’re about to walk into walls. Also if you’d be interested in a long-term technical collaboration for our migration, feel free to reply here or email me at phenakso@gmail.com.

6 comments

#learning-elixir #ai-agents

1 699 6

2026-08-21 11:28:03 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

manhvu

Have you built systems where processes live for days/weeks with accumulated state?

I have experience with this. For a GenServer with a long-lived session, it’s quite easy to keep the accumulated state in the process. However, if the GenServer crashes, it cannot recover its state if that state only exists in the process. In that case, the worker will simply restart with the initial state from init/1.

For simple cases, I would move the important state to an ETS table or some kind of database so that the process can recover it after a crash.

How do you handle hot code changes in production for stateful GenServers?

In production, this can be quite complicated because you need to handle state migration between OTP releases.

IMO, I would avoid running the application and Phoenix on the same VM/node. Instead, move the long-lived/stateful processes to another node so they can run independently from the Phoenix node.

Then you can use hot code upgrades more easily. When the state structure changes, you can migrate the existing state in the code_change/3 callback while upgrading the code.

Post #1

durvia

That’s very helpful.

btw I forget to attach the transcript when post, so here it is:

Transcript for selfdog demo

== selfdog spike ==
model: accounts/fireworks/models/deepseek-v4-pro-0813
provider: https://api.fireworks.ai/inference/v1
[driver] 12 controlled :hit calls sent since boot; Selfdog.Counter.read() = 15

=== user ===
The GenServer registered as Selfdog.Counter misbehaves.

Observation: since boot we sent it exactly 12 :hit calls and nothing else; its read() now returns 15.
A background workload now keeps sending :hit roughly every 300ms, so the system stays live while you work.

Diagnose the defect on the live node using your tools (inspect the process’s state, trace calls and returns),
then fix it WITHOUT restarting any process, then verify two things:

the counter’s accumulated value was preserved across your fix;
new :hit calls now increment the value exactly once each.

You cannot read the original source code, and you do not need it: diagnose from behavior alone,
then hot_patch with a complete, correct module of your own that is API- and state-compatible
(GenServer registered as Selfdog.Counter, state %{count: n}, hit/0 or hit/1 returning the new count, read/0 returning it).

Finish with a short written summary of the bug and your fix.

=== assistant (turn 1, reasoning) ===
Let me start by inspecting the process state and tracing calls.

[tool call, turn 1] observe_self %{“op” => “state”, “target” => “Selfdog.Counter”}
[tool result]
process: Selfdog.Counter pid<0.218.0>
state: %{count: 25}
info: [
memory: 6920,
reductions: 490,
message_queue_len: 0,
current_function: {:gen_server, :loop, 5}
]

[tool call, turn 1] observe_self %{“op” => “ps”}
[tool result]
top processes by reductions:
pid<0.95.0> reds=719633 mem=54KB q=0 cur={Process, :info, 2}
pid<0.50.0> reds=495979 mem=1761KB q=0 cur={:code_server, :loop, 1}
pid<0.228.0> reds=201768 mem=225KB q=0 cur={:gen_statem, :loop, 3}
pid<0.45.0> reds=147942 mem=29KB q=0 cur={:gen_server, :loop, 5}
pid<0.53.0> reds=15665 mem=416KB q=0 cur={:gen_server, :loop, 5}
pid<0.11.0> reds=14045 mem=5KB q=0 cur={:erl_prim_loader, :loop, 3}
pid<0.150.0> reds=13884 mem=21KB q=0 cur={:gen_server, :loop, 5}
pid<0.205.0> reds=13168 mem=2KB q=0 cur={:gen_server, :loop_hibernate, 4}
pid<0.0.0> reds=6284 mem=21KB q=0 cur={:init, :boot_loop, 2}
pid<0.49.0> reds=5793 mem=3KB q=0 cur={:gen_server, :loop_hibernate, 4}
pid<0.104.0> reds=4240 mem=16KB q=0 cur={:gen_server, :loop, 5}
pid<0.10.0> reds=3985 mem=5KB q=0 cur={Kernel.CLI, :exec_fun, 2}

supervision tree:

:workload pid<0.219.0> worker
Selfdog.Counter pid<0.218.0> worker

=== assistant (turn 2, reasoning) ===
The state is %{count: 25}. We sent 12 hits, but read returns 15 (at observation time). Now state is 25 due to background workload. Let me trace the hit function to understand the defect.

Let me trace Selfdog.Counter.hit.

[tool call, turn 2] observe_self %{“duration_ms” => 3000, “function” => “hit”, “module” => “Selfdog.Counter”, “op” => “trace”}
[tool result]
captured 40 Selfdog.Counter.hit events:
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 38 [pid<0.219.0>]
RET Selfdog.Counter.hit → 38 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 39 [pid<0.219.0>]
RET Selfdog.Counter.hit → 39 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 41 [pid<0.219.0>]
RET Selfdog.Counter.hit → 41 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 42 [pid<0.219.0>]
RET Selfdog.Counter.hit → 42 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 43 [pid<0.219.0>]
RET Selfdog.Counter.hit → 43 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 45 [pid<0.219.0>]
RET Selfdog.Counter.hit → 45 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 46 [pid<0.219.0>]
RET Selfdog.Counter.hit → 46 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 47 [pid<0.219.0>]
RET Selfdog.Counter.hit → 47 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 49 [pid<0.219.0>]
RET Selfdog.Counter.hit → 49 [pid<0.219.0>]
CALL Selfdog.Counter.hit() [pid<0.219.0>]
CALL Selfdog.Counter.hit(Selfdog.Counter) [pid<0.219.0>]
RET Selfdog.Counter.hit → 50 [pid<0.219.0>]
RET Selfdog.Counter.hit → 50 [pid<0.219.0>]

[tool call, turn 2] observe_self %{“duration_ms” => 3000, “function” => “handle_call”, “module” => “Selfdog.Counter”, “op” => “trace”}
[tool result]
captured 18 Selfdog.Counter.handle_call events:
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.858>]}, %{count: 50}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 51, %{count: 51}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.859>]}, %{count: 51}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 53, %{count: 53}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.860>]}, %{count: 53}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 54, %{count: 54}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.861>]}, %{count: 54}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 55, %{count: 55}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.862>]}, %{count: 55}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 57, %{count: 57}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.863>]}, %{count: 57}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 58, %{count: 58}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.864>]}, %{count: 58}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 59, %{count: 59}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.866>]}, %{count: 59}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 61, %{count: 61}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.867>]}, %{count: 61}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 62, %{count: 62}} [pid<0.218.0>]

=== assistant (turn 3, reasoning) ===
The trace reveals the bug clearly. Look at handle_call:

CALL handle_call(:hit, …, %{count: 50}) → RET {:reply, 51, %{count: 51}} — increments by 1. Good.
CALL handle_call(:hit, …, %{count: 51}) → RET {:reply, 53, %{count: 53}} — increments by 2! Bug.
count 53 → 54 (+1)
54 → 55 (+1)
55 → 57 (+2)
57 → 58 (+1)
58 → 59 (+1)
59 → 61 (+2)
61 → 62 (+1)

So sometimes it increments by 2. The pattern: it increments by 2 when count is odd? Let’s check: 51->53 (odd->odd, +2), 55->57 (+2), 59->61 (+2). And 50->51 (+1), 53->54 (+1), 54->55 (+1), 57->58 (+1), 58->59 (+1), 61->62 (+1).

So when count is odd, it increments by 2. When even, by 1. Actually 51 is odd → +2. 53 odd → +1? Wait 53->54 is +1. Hmm.

Let me list:
50 (even) → 51 (+1)
51 (odd) → 53 (+2)
53 (odd) → 54 (+1)
54 (even) → 55 (+1)
55 (odd) → 57 (+2)
57 (odd) → 58 (+1)
58 (even) → 59 (+1)
59 (odd) → 61 (+2)
61 (odd) → 62 (+1)

So odd → sometimes +2, sometimes +1. 51->+2, 53->+1, 55->+2, 57->+1, 59->+2, 61->+1.

Pattern: odd numbers 51, 55, 59 (i.e., count ≡ 3 mod 4) increment by 2. Odd numbers 53, 57, 61 (≡ 1 mod 4) increment by 1.

Actually 51 mod 4 = 3, 55 mod 4 = 3, 59 mod 4 = 3. 53 mod 4 = 1, 57 mod 4 = 1, 61 mod 4 = 1.

So the bug: when count mod 4 == 3, it increments by 2. This is a classic bug like if rem(count, 4) == 3, do: count + 2, else: count + 1 or something. Or maybe it’s a bitwise thing.

Actually a common bug: count + 1 + (count &&& 2)? No. Let me think: count &&& 2 gives 2 when bit 1 set (count mod 4 in {2,3}), 0 otherwise. So count + 1 + (count &&& 2) would give: count mod 4 = 3 → +3? No.

Let me think of count + 1 + rem(count, 2)? That would be odd → +2, even → +1. But 53 odd → +1, not +2. So no.

Actually maybe the bug is something like: new = count + 1; if rem(new, 4) == 0, do: new + 1, else: new. Let’s check: count=51 → new=52, rem(52,4)=0 → 53. Yes +2. count=53 → new=54, rem=2 → 54. +1. count=55 → new=56, rem=0 → 57. +2. count=57 → new=58 rem=2 → 58 +1. count=59 → new=60 rem=0 → 61 +2. count=61 → new=62 rem=2 → 62 +1.

Yes! So the bug is: after incrementing, if the new count is divisible by 4, it increments again. Or equivalently, it skips multiples of 4.

Actually wait, let me reconsider. The exact bug doesn’t matter too much; I just need to write a correct module. The correct behavior: hit increments by exactly 1.

But I should understand enough to write a correct module. The correct module:

defmodule Selfdog.Counter do
  use GenServer

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, %{count: 0}, opts)
  end

  def hit(server \\ __MODULE__) do
    GenServer.call(server, :hit)
  end

  def read(server \\ __MODULE__) do
    GenServer.call(server, :read)
  end

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_call(:hit, _from, %{count: n} = state) do
    {:reply, n + 1, %{state | count: n + 1}}
  end

  @impl true
  def handle_call(:read, _from, state) do
    {:reply, state.count, state}
  end
end

Wait, but I need to preserve the current count (25 at time of state inspection, but it’s been incrementing). The hot_patch replaces the module code but keeps process state. So the state %{count: n} is preserved. Good.

But wait - I need to check the API. The trace shows hit() and hit(Selfdog.Counter) both being called. So hit/0 and hit/1. Also read/0. Let me check read. The observation says read() returns 15. Let me check what read does.

Actually I should check the read function too. But I can just implement read/0 and read/1.

Let me also verify: the trace shows hit() calls hit(Selfdog.Counter). So hit/0 delegates to hit/1 with default server name.

Now, importantly, the current count. The state was 25 when I inspected, but it’s been incrementing. The hot_patch preserves state, so whatever the current count is will be preserved. Good.

Let me write the correct module. I need to make sure the module is complete and correct.

Actually, let me also double check there’s no other subtlety. The bug is “increments by 2 sometimes”. The fix is to always increment by 1.

Let me also consider: is there a read function that might also be buggy? The observation says read() returns 15 after 12 hits. That’s consistent with the +2 bug (12 hits, some +2, total 15). So read is fine, just returns count.

Let me write the patch now.

Actually, let me verify the current count right before patching to confirm preservation. I’ll patch then read.

Let me write the complete module.

=== assistant (turn 3) ===
The trace reveals the bug clearly. Look at the handle_call returns:

%{count: 50} → 51 (+1)
%{count: 51} → 53 (+2)
%{count: 53} → 54 (+1)
%{count: 55} → 57 (+2)
%{count: 59} → 61 (+2)

The pattern: whenever the new count lands on a multiple of 4, it gets bumped again (51→52→53, 55→56→57, 59→60→61). So the counter skips multiples of 4, incrementing by 2 in those cases.

Let me capture the current count right before patching, then apply the fix.

[tool call, turn 3] observe_self %{“op” => “state”, “target” => “Selfdog.Counter”}
[tool result]
process: Selfdog.Counter pid<0.218.0>
state: %{count: 133}
info: [
memory: 11808,
reductions: 2233,
message_queue_len: 0,
current_function: {:gen_server, :loop, 5}
]

=== assistant (turn 4, reasoning) ===
Current count is 133. Let me patch with a correct module.

[tool call, turn 4] hot_patch %{“module” => “Selfdog.Counter”, “source” => “defmodule Selfdog.Counter do\n use GenServer\n\n def start_link(_opts \\ ) do\n GenServer.start_link(MODULE, %{count: 0}, name: MODULE)\n end\n\n def hit do\n GenServer.call(MODULE, :hit)\n end\n\n def hit(_server) do\n GenServer.call(MODU@implimplimplimplE, :hit)\n end\n\n def read do\n GenServer.call(M@implDULE, :@implead)\n end\n\n @implimpl true\n def init(state) do\n {:ok, state}\n end\n@impln @impl true\n def handle_call(:hit, @implfrom, %{cou@implt: n} = state) do\n {:reply, n + 1, %{state | count: n + 1}}\n e@impld\n\n @impl true\n def handle_call(:read, _from, state) do\n {:reply, state.count, state}\n end\nend\n”}
warning: redefining module Selfdog.Counter (current version loaded from _build/dev/lib/selfdog/ebin/Elixir.Selfdog.Counter.beam)
└─ selfdog.hot_patch.ex:1: Selfdog.Counter (module)

[tool result]
patched: Selfdog.Counter recompiled and loaded live; old version purged. Running processes keep their state and use the new code on the next call.

[tool call, turn 5] observe_self %{“op” => “state”, “target” => “Selfdog.Counter”}
[tool result]
process: Selfdog.Counter pid<0.218.0>
state: %{count: 151}
info: [
memory: 5776,
reductions: 2623,
message_queue_len: 0,
current_function: {:gen_server, :loop, 5}
]

[tool call, turn 5] observe_self %{“duration_ms” => 3000, “function” => “handle_call”, “module” => “Selfdog.Counter”, “op” => “trace”}
[tool result]
captured 20 Selfdog.Counter.handle_call events:
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1135>]}, %{count: 151}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 152, %{count: 152}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1136>]}, %{count: 152}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 153, %{count: 153}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1137>]}, %{count: 153}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 154, %{count: 154}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1138>]}, %{count: 154}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 155, %{count: 155}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1139>]}, %{count: 155}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 156, %{count: 156}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1140>]}, %{count: 156}) [reference<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 157, %{count: 157}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1141>]}, %{count: 157}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 158, %{count: 158}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1142>]}, %{count: 158}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 159, %{count: 159}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1143>]}, %{count: 159}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 160, %{count: 160}} [pid<0.218.0>]
CALL Selfdog.Counter.handle_call(:hit, {pid<0.219.0>, [:alias | reference<0.0.28035.20771626.411107329.1144>]}, %{count: 160}) [pid<0.218.0>]
RET Selfdog.Counter.handle_call → {:reply, 161, %{count: 161}} [pid<0.218.0>]

=== assistant (turn 6, reasoning) ===
The fix works. Every hit now increments by exactly 1, including through multiples of 4 (152→153, 156→157, 160→161). State preserved (151 at patch time, continued incrementing).

Let me verify read() returns correct value and confirm state preserved. The state was 133 before patch, then 151 at patch time (background workload kept incrementing). Actually I need to verify “accumulated value preserved across fix”. The value at patch time was 151 (from state check right after patch). It continued to 161. So preserved.

Let me do a final read to confirm.

[tool call, turn 6] observe_self %{“op” => “state”, “target” => “Selfdog.Counter”}
[tool result]
process: Selfdog.Counter pid<0.218.0>
state: %{count: 173}
info: [
memory: 5776,
reductions: 3128,
message_queue_len: 0,
current_function: {:gen_server, :loop, 5}
]

=== assistant (turn 7) ===
The fix is verified. Every handle_call(:hit, ...) now returns exactly count + 1 (152→153→154→155→156→157→158→159→160→161), with no more double-increments.

Summary

Bug: The hit handler incremented the counter by 2 whenever the resulting value would land on a multiple of 4. Tracing handle_call showed the pattern clearly: 50→51 (+1), 51→53 (+2), 53→54 (+1), 55→57 (+2), 59→61 (+2), etc. The counter was effectively “skipping” multiples of 4, which is why 12 :hit calls produced a value of 15 instead of 12.

Fix: I hot-patched Selfdog.Counter with a clean, correct implementation whose handle_call(:hit, ...) simply returns n + 1 and stores %{count: n + 1} — a single, unconditional increment. The module remains API- and state-compatible (GenServer registered as Selfdog.Counter, state %{count: n}, hit/0 and hit/1 returning the new count, read/0 returning it).

Verification:

State preserved — the process was never restarted; its accumulated count carried straight through the patch (133 at patch time, continuing to grow under the live workload).
Exact single increments — post-patch traces show every :hit call advancing the count by exactly 1 (e.g. 151→152→153→154→155→156→157→158→159→160→161), with no more +2 jumps.

Post #2

manhvu

It’s too long for me to read . Elixir process can process only one message at time then issue like counter that is increased by 2 is related your logic code only. I think better logging at handle_call with :hit action than get process’s state by tool call (I’m not sure about your case, but get state with interval or something like that is not perfect way to deal with state of process in concurrency).

Post #3

mikehostetler

This is exactly the kind of problem OTP was built to solve.

BEAM gives you strong process semantics, but durable agent state still needs explicit design.

A process can live for weeks. That’s normal. But supervisors restore processes, not accumulated state. Your append-only stream is a good source of truth. The GenServer state can be a fast projection that you rebuild after failure.

A few practical points:

Registry works well for local addressing. Distribution adds partitions, retries, duplicate delivery, and agent relocation.
GenServers serialize work, but mailboxes don’t provide automatic backpressure.
Phoenix and the agent runtime can share an OTP application. Separate them when scaling or failure boundaries require it.
Hot upgrades work, but state migration, rollback, and cluster coordination need careful design.

There’s also a growing Elixir AI stack:

LLMDB provides local model metadata for capabilities, limits, pricing, and providers. It’s now available as @agentjido/llmdb on NPM.
ReqLLM is the Elixir equivalent of Vercel’s AI SDK. It provides one interface for generation, streaming, structured output, tools, and multiple providers.
Jido is a full OTP-native agent framework. It covers agents, actions, signals, supervised processes, and multi-agent coordination.

Full Disclosure: I’m the author of Jido, ReqLLM, and LLMDB.The projects are separate layers. You can use ReqLLM without Jido and LLMDB without either.

Jido isn’t the only agent option:

I’d be glad to compare notes, especially around your event envelope, addressing model, trajectory storage, and recovery semantics.

Post #4

durvia

@mikehostetler Thanks for the detailed response. Happy to compare notes. I’ve read some of the ReqLLM and Jido source, so I can be specific here.

Addressing model

Our address is platform:kind:id. Platform decides how its subtree is organized and what it can do, kind decides the level in the tree. It serves as agent def unit, memory units belonging, routing key, supervision tree path, and storage partition key simultaneously. The hierarchy maps 1:1 to the supervision tree, so parent-dies-children-die comes free from OTP.

On distribution: we’re single-instance-per-org for now. Your warning about partitions, retries, and agent relocation is helpful. That’s the part we haven’t had to solve yet.

Event envelope and admission

Every inbound is wrapped into a message that passes through a gate before reaching the agent. The gate stamps it with metadata (authority, integrity, etc.) and makes a ruling: admit (interrupt/wake the agent now), follow (land after the current turn), park (hold for the next turn), or refuse. The agent receives admitted messages as a batch per call/action boundary (per call for not urgent ones, per action for steer: after thinking, after text output, after one single tool call).

On top of this we have a structured cross-agent protocol for peer-to-peer communication, currently built in TS, though OTP holds this natively. It includes wake judges (code + LLM decide should it wake the agent or wait for batch), read receipts, and bounce-on-failure.

The gate gives us structured triage, but the queue can still grow unboundedly under load. Have you seen good backpressure patterns in production, or does this have to be handled at the inbound layer specifically?

Trajectory storage and recovery

Every model interaction appends to a per-address event stream as it happens, one file per address. The state is a projection rebuilt from that stream. Crash recovery, restart, swapping which agent definition occupies an address, all the same path: load the definition seed, replay the stream, arrive at current state.

We also need the stream as a full audit trail (what the agent saw, what it produced, under what configuration), so we went with append-on-every-interaction instead of checkpoint-on-idle. Curious how checkpoint recovery has played out for you in practice, especially around long-running agents where the checkpoint-to-crash window matters.

ReqLLM

We’re planning to use ReqLLM as our LLM client layer. The provider coverage and the Req plugin architecture are what we need, so we want to build on it instead of rolling our own.

I noticed the 2.0 roadmap (#830) is parked for now. The change we’re most interested in is #2, canonical tagged events becoming the default streaming contract. We’re currently building against StreamChunk, so knowing the rough timeline for that transition would help us decide whether to target the new event surface from the start or bridge later.

Jido

You contribution is a great study material for us. And the StateOp vs Directive type separation is a clean vocabulary, the hook phase permission narrowing (handle_signal → prepare_action) is something we’re adopting. We made different choices in a few places (runtime-mutable composition, supervision tree as topology, Saga-style compensation), but we are still early in migration, so some of these may converge.

Post #5

mikehostetler

Thanks for the detail. The idea behind your “gate” feels like the key abstraction here. admit / follow / park / refuse maps well to OTP.

For backpressure, I’d handle it before work reaches the agent mailbox. Use a bounded inbox per address, then apply your explicit policy there. Take a look at how Broadway or GenStage handle this, they are great examples. Keep in mind that OTP mailboxes give isolation, but not backpressure.

Your event stream model also looks sound. I’d treat checkpoints as replay optimizations / projections, not another source of truth. Store the stream offset with each checkpoint, then replay the tail after recovery.

ReqLLM 2.x is exploratory, with no fixed timeline. If canonical tagged events are useful now, I’m happy to accept a PR that brings an additive event surface into 1.x. What minimum event types and lifecycle guarantees does your runtime need? Happy to collab here. Even if you want to toss up an issue with more detail, I’m happy to chew on it together. I’m guessing you’re doing this already, but llmdb is a critical part of ReqLLM. If you haven’t dug into that yet, it’s worth a deep dive.

The differences from Jido are also interesting - especially runtime-mutable composition and supervision as topology.

Sounds like you’re on the right track!

Post #6

— All posts loaded —

Where Next?

View thread on forum (has 6 replies!)

learning-elixir

ai-agents

Home Chat & Discussions>Discussions

8 444 6

Last post

REPLY VIEW THREAD WITH 6 POSTS

Building an AI agent runtime moving from TypeScript to Elixir/OTP - can you help/compare notes?

durvia

How I ended up here

What I’ve tried

My Questions

Also looking for collaborators

Showing Posts 1 to 6 Show Best PostsShow All (oldest first)Show All (newest first)

manhvu

durvia

Summary

manhvu

mikehostetler

durvia

Addressing model

Event envelope and admission

Trajectory storage and recovery

ReqLLM

Jido

mikehostetler

Where Next?

Trending in Discussions

What Elixir related stuff are you doing?

My thoughts on Ash

Cure - a new language compiled to BEAM

Can the ideas from DeepSeek Harness be transferred to Elixir eco?

Seeking feedback on my first Phoenix project - a real-time, multiplayer online version of the Scattergories game

Companies that hire candidates without prior Elixir experience

Building an AI agent runtime moving from TypeScript to Elixir/OTP - can you help/compare notes?

Other Trending Topics

Hobbes - a low-level distributed database for the Elixir programming language

Gust - A task orchestration system built in Elixir

Beam Bots - Resilient Robotics on the BEAM

ExRatatui - Elixir bindings for the Rust ratatui terminal UI library

Emerge & Solve - a GUI framework for Elixir

Corex - Accessible and unstyled UI Phoenix components

Chat & Discussions>Discussions

Latest on Elixir Forum

Sponsor Spotlight

Our Sponsors

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

We're in Beta

Options

Thread Display Mode

Thread Preview

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)