@ausimian@polvalente — both of your runs landed on main this afternoon. Quick summary of what shipped, because the rollup matters more than the diff.
We now have a first-class --runtime-profile emily lane. The :emily clause in RuntimeProfile.resolve/1 mirrors :emlx’s Apple-shaped flags but routes to Emily.Backend and ships the empirical per-profile margin floors directly from ausimian’s validation pass — agent worst 0.417 on two_assistant_turns, role worst 1.029 on escalate_to_human, 80% rule, so agent: 0.33 / role: 0.82. The previous {:custom, Emily.Backend, []} recipe still works for one-off overrides, but the day-to-day operator path is just:
No --min-*-margin flags needed; the profile carries them. Paulo, your bare-EMLX 37/37 result is now the reference for :emlx; ausimian’s bare-Emily 36/37 + per-profile-floor result is the reference for :emily. Both lanes are decision-stable against the CUDA snapshot (agent_id, role_id, token_count, transcript_hash all match); route_hash drifts on every case for both, which is what the per-profile snapshot fixture lane below is designed for.
A few other things that landed in the same push, attributed: ausimian’s “honest per-tensor SVD timings” follow-up became Phase 1 (Exporter.sync_tensor!/1 is a one-line backend-agnostic force point inside timed_decompose/2 + timed_reconstruct/3 — closes the decompose_elapsed_ms: 2 artefact for any lazy backend, costs ~nothing on EXLA). Paulo’s runtime.ex:56 review nudge became Phase 2 (tensor_backend/1 default is now tensor.data.__struct__ |> Module.split() |> Enum.join(".")) plus a Phase 3 audit that found two more sites in Sakana.{Artifact, PythonImporter} doing the same brittle string-cond — they’re consolidated behind Runtime.BackendLabel.from_label/1 now, with EMLX label coverage that the prior :emlx lane was actually missing.
Per-profile snapshot fixtures: drop examples/fixtures/runtime_profiles/emily/qwen_router_prompt_eval_logits.json and the eval picks it up automatically when --runtime-profile emily is passed without an explicit --snapshot. The seed run is mix run examples/qwen_router_prompt_eval.exs --runtime-profile emily --snapshot-out <path>. If anyone wants to send me the Emily-side rounded logits + route_hashes for all 37 cases, I’ll commit it as priv-clean and the next person to try this on Apple gets a stable baseline to compare against. Otherwise it stays unseeded and the snapshot drift assertion is just skipped on Emily, same way it is today on CUDA when no --snapshot is passed.
Re ausimian’s FastKernels follow-up commit: yes please, but as a separate :emily_fast profile rather than wired into :emily’s Coordinator.load/1 directly. That keeps bare-Emily as the reference research lane (matches Paulo’s bare-EMLX baseline) and makes the FastKernels effect attributable to its own dependency version — important if a future Emily release shifts behaviour and we have to bisect. The 15% wall-clock win on a single forward pass is real and should be considerably bigger on a decode loop. Detailed reply on the PR itself.
Thanks both — the diagnosis-to-merge round-trip on this was faster than I had any right to expect for an Apple-Silicon lane I can’t run myself.
I want to zoom out a bit and write down where I think this is heading architecturally, because the recent progress changes the nature of the problem.
The thread started as “can Sakana AI’s TRINITY/Qwen router be ported to Elixir/Bumblebee/Nx/Axon at all?” The answer is now much closer to yes than it was a few weeks ago. The thin-SVD work in Nx PR #1753 removes the worst memory pressure from export. We have CUDA as the original reference lane, EMLX getting a clean 37/37 decision-stable pass, and Emily now close enough that it has a real runtime profile with its own empirical margins. The remaining differences are the kind I would expect from backend-specific numeric behavior: route hashes drift, but the actual routing decisions can be made stable with per-profile fixtures and margin floors.
That is a big deal. The “can Nx express this pipeline?” question is mostly behind us. The harder question now is where this code should live before the bring-up repo hardens into the permanent monolith.
Right now trinity_coordinator is doing too many jobs. That was the right way to get the port working, but it is the wrong long-term shape.
It currently contains, in one application:
the TRINITY coordination semantics
the Sakana/Qwen manifest and routing-head invariants
Bumblebee model loading
Axon routing-head execution
Nx/SVD export and reconstruction mechanics
safetensors slicing
artifact fetching and pinning
provider-agent pool wiring
trace plumbing
operator Mix tasks
runtime profile and XLA/EXLA/EMLX/Emily bring-up logic
As a lab bench, that is useful. As a library boundary, it is too heavy.
The concrete split I am converging on is:
Product / AppKit
|
| governed product boundary
v
Mezzanine / OuterBrain / Citadel
|
| semantic, workflow, authority, review, provenance
v
trinity_framework
|
| Trinity.* contracts and coordinator semantics
|
+--> trinity_bridge_inference
| |
| v
| inference
| provider LLM calls
|
+--> trinity_bridge_self_hosted_inference
|
v
self_hosted_inference_core
|
v
self_hosted_inference_bumblebee
|
| Qwen load + Sakana adapter patch + routing head projection
v
RouteLogits
Reusable ML substrate underneath:
crucible_safetensors
crucible_tensor_patch
crucible_factorization
crucible_model_registry
The important point is that trinity_framework already exists and already has real Trinity.* contract modules. There is also already a consumer using it in stack_lab’s examples/trinity_platform_roundtrip. The workspace scaffold is there: core/, bridges/, apps/, tools/. Most of those package slots are stubs today, but the shape is right.
So I would avoid turning trinity_coordinator into a bigger framework. The cleaner next step is to finish the framework that already exists, move the reusable pieces to their real owners, and let trinity_coordinator become a temporary reference deployment while the migration is happening.
The main boundary correction is that the coordinator should never see the hidden vector.
An earlier version of the design had this rough flow:
Qwen runtime extracts hidden state
-> returns HiddenVectorRef or hidden-state payload
trinity_sakana_pipeline applies routing head
-> route decision
That split puts pressure in the wrong place. It either transfers a large tensor over a boundary or makes the coordinator orchestrate a runtime-local tensor operation through an opaque handle. Both are awkward. The extraction and the routing-head projection belong together inside the model runtime.
The cleaner flow is:
trinity_sakana_pipeline builds a plan
-> artifact pin
-> adapter ref
-> selected tensor/head spec
-> TRINITY shape invariants
self_hosted_inference_bumblebee executes the runtime work
-> load Qwen
-> apply/cache Sakana adapter
-> extract penultimate state
-> run Axon routing head
-> return small RouteLogits struct
trinity_coordinator_core applies semantics
-> role/agent partition
-> Worker/Thinker/Verifier choice
-> RouteDecision
That keeps tensors where tensors belong. The model runtime owns Bumblebee, Axon, Nx backend placement, and the routing-head math. The coordinator core owns what the logits mean.
The same idea applies to adapter identity. Sakana’s patch differs from LoRA in details, but operationally it behaves like an adapter: a content-addressed parameter-efficient modification that should be reused if the same patch is requested again. I would rather make that visible in the runtime lease model than hide it inside the Bumblebee backend as an ad hoc cache. self_hosted_inference_core should understand it at the lease level.
Concretely, SelfHostedInferenceCore.InstanceSpec should grow an optional adapter_ref, and RuntimeRegistry should be able to route by something like:
(backend_id, adapter_ref)
instead of only by backend/profile. That lets an already-warmed Qwen+Sakana-adapter runtime handle the next request with the same adapter hash. Without that, every distinct TRINITY patch plan risks becoming a fresh GPU process or a contested in-place mutation.
This is the part of the migration I expect to be hardest. Adding adapter_ref may be additive, but it may also expose assumptions in the existing Ollama backend or runtime registry about what “instance identity” means. I would wait for the real CUDA head_route gate to pass through the new backend before calling this a casual field addition.
The ML utility split is also important. The Nx PR #1753 point is a good example. One operation, Nx.LinAlg.svd, currently forces a git-SHA pin through the whole coordinator because export needs the better thin-SVD memory behavior. That tradeoff is fine in a bring-up repo. It becomes a problem when downstream consumers only want TRINITY routing semantics.
The SVD/SVF code should move to crucible_factorization. That one package can carry the Nx pin while it needs to. When the relevant Nx release lands on Hex, that package can relax the dependency. Other packages should be able to depend on a RouteDecision struct without inheriting a custom Nx commit.
One correction to my earlier thinking: I would fold this into crucible_model_registry instead of creating a new crucible_artifacts package. The registry already exists in the North-Shore-AI namespace and already has the richer storage-backend concept. The coordinator’s artifact pin schema should be merged into that. But that needs a real pre-flight audit. If the registry’s notion of artifact identity, SHA scope, revision refs, or cache layout differs from TrinityCoordinator.ArtifactFetch.Pin, then the work becomes schema reconciliation instead of a module rename.
Inside trinity_framework, I would split the TRINITY pieces roughly like this:
core/trinity_contracts
DTOs and behaviours:
RouteDecision, RouteLogits, AgentCallIntent, ProviderPool contracts,
trace events, model-runtime behaviour, agent-caller behaviour.
No Nx, no Axon, no Bumblebee, no Req, no HF.
core/trinity_sakana_contracts
Sakana/Qwen schemas and invariants:
manifest shape, router-head dimensions, export spec,
selected tensor keys, profile specs.
Still no runtime deps.
core/trinity_sakana_pipeline
TRINITY-specific plan generation:
what adapter/head/tensor plan should be executed.
It returns a plan rather than a hidden vector or model handle.
core/trinity_coordinator_core
the pure coordination state machine:
run loop, role injector, thinker/verifier policy,
route-logit interpretation, budget enforcement.
It should be testable with mock behaviours and no real model.
bridges/trinity_bridge_self_hosted_inference
translates Trinity.ModelRuntime calls into self_hosted_inference_core leases
and self_hosted_inference_bumblebee.route_with_head/3.
bridges/trinity_bridge_inference
provider LLM calls through the existing :inference package.
bridges/trinity_bridge_trace
trace sinks and redaction policy.
tools/trinity_ops
the Mix tasks currently under mix trinity.*
apps/trinity_single_node
the production-shaped local deployment app.
That last package matters because the long-lived deployment surface should move out of trinity_coordinator. The app that wires the single-node experience should live under the trinity_framework workspace as apps/trinity_single_node. It should own the runtime config reads: XLA_TARGET, HF_TOKEN, provider keys, runtime profile selection, artifact dirs, and so on. Those values should be materialized at the app boundary and passed down as config, so lower libraries receive config instead of reaching into env directly.
During migration I would keep the old coordinator config authoritative until the new app proves equivalence. In practice that means running the same HITL head-route gate first against trinity_coordinator’s config/runtime.exs, then against trinity_single_node’s config/runtime.exs. Only after both pass should the original config be retired.
There are also some boring but necessary enforcement rules:
No System.get_env under lib/**
Runtime/deployment env reads stay in config/runtime.exs or app bring-up.
No Nx/Axon/Bumblebee in contracts
Contract packages must remain lightweight and CPU/node friendly.
No direct SelfHostedInferenceCore/Bumblebee/Crucible imports in core packages
Those imports belong in bridges, runtimes, or ML utility packages.
No product bypass
Product code should go through AppKit/Mezzanine/Trinity surfaces,
not directly into lower runtime or ML packages.
The last point ties this back to the older governed execution platform thread:
That thread is out of date in details, but the boundary argument still applies. The larger stack has separate places for semantic reasoning, authority, workflow, lower facts, traces, review, and execution because those are different kinds of truth. TRINITY should plug into that shape rather than bypass it.
So the decomposition goes beyond moving files into smaller repos. It makes the TRINITY port obey the same truth boundaries as the rest of the platform.
The proposed phase order is intentionally conservative:
0. Freeze the monolith baseline.
1. Extract crucible_safetensors first, as the smallest proof of the pattern.
2. Extract crucible_factorization and move the Nx #1753 pin there.
3. Extract crucible_tensor_patch and merge artifact pins into crucible_model_registry.
4. Build self_hosted_inference_bumblebee and prove route_with_head/3 on CUDA.
5. Fill out trinity_framework core and bridges.
6. Move operator Mix tasks into tools/trinity_ops.
7. Make apps/trinity_single_node the deployment app.
8. Reduce trinity_coordinator to a compatibility shim.
9. Archive it after a release window.
The file audit gives this some teeth. The coordinator has about 60 .ex files under lib/, about 10.5 KLOC total. Roughly 40% belongs in trinity_framework, 30% in self_hosted_inference_bumblebee plus the North-Shore-AI crucible_* packages, 25% in ops/tasks, and a small amount should be deleted because provider transport already belongs in inference.
Two examples show the logic:
TrinityCoordinator.Sakana.Artifact is currently three owners in one file: manifest validation, artifact fetch/pin semantics, and model-state patching. Those should become:
TrinityCoordinator.CoordinationHead is also split in two:
role/agent/head invariants -> trinity_sakana_contracts
Axon model and projection -> self_hosted_inference_bumblebee
That split is the whole design in miniature: contracts define what is valid, runtime executes tensor work, coordinator interprets small semantic results.
I also want to preserve the git history when this moves. The history on files like sakana/svd.ex, coordination_head.ex, and the Sakana loader/patch code contains a lot of architectural decision-making that will not be obvious from the final file layout. Before trinity_coordinator is retired, I would subtree-merge or otherwise import the relevant history into trinity_framework.
My current summary would be:
trinity_coordinator was the right place to get the port working. The permanent library boundary should move elsewhere.
The reusable ML pieces should become North-Shore-AI crucible_* packages, with the existing crucible_model_registry extended rather than duplicated.
The single-node app should live as trinity_framework’s apps/trinity_single_node.
And the old coordinator repo should become a temporary compatibility shim, then retire.
If this sounds like a lot of decomposition for one model port, I think the recent backend work is exactly why it is justified. The port now spans CUDA, EMLX, and Emily lanes; it carries backend-specific profile behavior; it depends on a very specific Nx SVD improvement; and it is starting to look useful beyond the initial experiment. That is the point where “just keep it in the bring-up repo” starts creating future debt.
The goal is to avoid abstraction for its own sake and give each part one honest owner:
contracts describe
coordinator semantics decide
runtime executes tensor work
crucible packages do reusable ML mechanics
ops tasks operate
single-node app configures
AppKit/Mezzanine/OuterBrain/Citadel/Jido keep the governed platform honest
I can release that PR on a patch release soon (e.g. this weekend) instead of holding it off til 1.0. Also, Hex won’t allow git dependencies at all, so splitting the dependency off doesn’t help much.
Our custom Qwen3 implementation now reaches 300 tok/s on 0.6B 4bit and the emlx_axon bumblebee rewrites can reach 120 tok/s on the non-quatized 0.6B! Tomorrow I’ll release a new emlx/emlx_axon version with these improvements. Improvements are available in emlx/emlx_axon 0.4!
