Adding a UI for Squid Mesh (read-only for now):
webofbits
3
webofbits
Once I finish the remaining long-running step recovery work, I’ll feel much more confident moving Squid Mesh from alpha toward beta.
The main architectural gap I still want to close is the runtime switchover to the Jido-native coordination path. The direction is to make workflow and dispatch agents rebuild from durable journal state, with claim fencing, leases, heartbeats, completion, failure, retry, and recovery all represented as durable facts.
For long-running steps specifically, the missing piece is heartbeat-backed recovery. I’m planning to lean on the model from Mike Hostetler’s Intent Ledger project: GitHub - mikehostetler/intent_ledger · GitHub . Once Squid Mesh can safely recover claimed long-running work after worker crashes, restarts, or expired leases, the runtime story should be solid enough that I’d be comfortable calling the project beta rather than alpha.
Still a few focused PRs away, but the pieces are starting to line up.
0
dimitarvp
OMG this is awesome. I already feel like I am missing out though I know you are not quite done with the initial items we discussed (or at least this is what I extracted from the recent announcements).
0
webofbits
Yeah. We’ll get there. Thanks for all the hints and feature suggestions. Hopefully the last 9 issues are what we need to call it a solid v1 wrap! Issues · dark-trench/squidie · GitHub
1
ghannam80
Thank you for the great addition to elixir ecosystem.
I have a question why you opted to have it running inside the related parent application instead of having it running independently as in my work most of the times we need durable workflow for processes that spans across multiple systems and teams?a workflow could be triggered through API or UI form or even timed event ,..etc then it start interacting with people or related systems based on defined flow rules.
By the way do you plan to add SLA capability per each step ? Also , capability to limit what each user see in the flow as sometimes you have external users who should not all internal steps and communication
0
webofbits
Thanks, appreciate the thoughtful questions.
The main reason Squid Mesh is designed to run inside the host application is proximity to domain logic. Most workflow steps need to call existing contexts, use the app’s auth/authorization rules, share the same database transaction boundaries, emit domain events, and reuse local observability. If the workflow runtime runs as a separate service, you often end up rebuilding a second integration layer just so it can call back into the system that already owns the business rules.
That said, it does not have to be embedded in a large existing app. You can absolutely create a small dedicated Elixir app that embeds Squid Mesh and acts as the workflow host for a broader system. That is probably the right shape when a workflow spans multiple teams or systems: keep Squid Mesh close to the orchestration/domain code for that workflow, while integrations with other systems happen through APIs, messages, webhooks, etc. We also have a standalone harness today, but it is mostly for QA/smoke testing rather than the recommended production deployment shape.
On SLA per step: if you mean things like expected completion time, timeout/deadline, escalation, alerting, or “this approval must happen within 24h”, then yes, that fits the direction of the project. Some of that can already be modeled with retries, pauses/manual actions, and external monitoring, but first-class step-level deadlines/escalations would be a good capability.
On permissions/visibility: yes, I think that is important. The runtime should be able to expose different views of the same run depending on the actor. For example, an external user may see only their task/status while internal operators see full step history, errors, audit events, and system communication. That likely belongs as an authorization/read-model layer rather than changing the underlying workflow history.
0
webofbits
@dimitarvp FYI: this is the summary of features implemented after your suggestions:
Implemented:
- Human-in-the-loop pause/manual unblock:
:pause, :approval, unblock_run/2, approve_run/3, and reject_run/3. - Machine-readable explanation/introspection:
SquidMesh.explain_run/2 returns reason, evidence, and next actions. - Saga compensation callbacks:
compensate: SomeStep runs completed reversible steps in reverse completion order and persists recovery metadata. - Irreversible/non-compensatable markers:
irreversible: true and compensatable: false affect recovery policy and block replay unless allow_irreversible: true. - Compensation vs undo distinction:
recovery: :compensation | :undo exists on :error transitions and is surfaced in docs/audit history. - Accumulated state access:
Step input can merge payload, run context, and completed dependency outputs, with input: and output: mapping. - Static fan-out/fan-in:
Dependency workflows support after: […], multiple root steps, joins, and independent host-executor dispatch. - Local transaction grouping:
transaction: :repo wraps a single custom step callback in the configured Ecto repo transaction. - Positioning against adjacent libraries:
Docs compare Squid Mesh with Sage, Reactor, FlowStone, Runic, and Jido. - Squid Mesh-native step API:
SquidMesh.Step is now the preferred authoring contract; raw Jido.Action remains an interop
path.
Remaining or partial:
- Full compensation/undo/rollback semantics:
Compensation exists, undo is represented as recovery routing metadata, but there is not yet a
full automatic undo-chain engine with partial-success reporting. - Rich conditional/deferred continuation:
Basic accumulated state exists, but KYC-style “defer this step and continue/poll later” behavior is still planned. - Dynamic step injection / graph expansion:
Still planned, not implemented. - Transaction across fan-out/sub-steps:
transaction: :repo only covers one local step callback, not multiple workflow steps or distributed work. - Long-running step recovery:
Lease/heartbeat/fencing foundations exist in the Jido-native path, but heartbeat-backed live recovery is not fully wired yet. - Journal-backed inspection/explanation:
Current inspection/explanation reads Postgres runtime tables; durable journal/checkpoint projections are still open work. - Full Jido-native runtime switchover:
Protocols, journals, workflow agents, and dispatch agents exist as foundations, but the live
runtime still uses the current Postgres plus host-executor path. - Exactly-once external effects:
Explicitly out of scope; external systems still need idempotency keys or duplicate-safe behavior.
1
webofbits
I’ve been thinking that Squid Mesh is already a strong use case for Jido, but it probably also needs its own strong use case.
One idea I’m considering is Rift: an embeddable Phoenix/LiveView ops inbox for workflows that need human decisions.
The rough idea is that the host app defines case types in code. Users open cases through host-configured forms. Each case starts a Squid Mesh workflow run. Operators then review cases in a LiveView inbox, claim/assign them, approve/reject/cancel, and inspect runtime details through SquidSonar.
The goal would not be to build a generic help desk or a no-code workflow builder. The host app would still own domain logic, auth, users, tenants, teams, file storage, and the actual workflow modules. Rift would provide the human-facing ops surface around durable workflows.
Part of the motivation is dogfooding. A real app like this would put pressure on Squid Mesh in useful places: human-in-the-loop flows, approvals/rejections, cancellation, audit trails, side effects, failed side effects, stale operator actions, and runtime inspection. That seems like a better feedback loop than only adding examples inside the main repo.
I started a PR with a PLAN.md: GitHub - ccarvalho-eng/rift: LiveView ops inbox for human workflow decisions · GitHub
Curious if this direction resonates with people who have had to build internal approval queues, ops review flows, or human-in-the-loop admin workflows in Phoenix apps.
3
dimitarvp
Dogfooding absolutely is the best way to test your library in the real space without waiting for your users to do so. Kudos.
1
webofbits
Yeah. I wanted to be proactively battle testing it against some real use-cases.
Excited to fill the main gap: recover of long running/stale steps. Once that’s in place, and it’s using Jido journals for persisting and reconstructing runs I’d say the work on the alpha should be settled.
0