Llamacpp, mlx-lm and derivatives. All via openai protocol. Different ports/hosts, no api key. Still experimenting (experimental codegen with small local models. My current experiment went from reqllm back to req. it seemed easier to experiment with tool call formats and slighly broken json directly than through reqllm. That was claude codes’ opinion though , so take it with a large dose of salt 
).
I will check out your project later, tracking experiments over time is useful.
I’ve been thinking about a next step for Aludel: adding a Build → Evaluate → Export workflow. The idea is to let people design agentic workflows visually, use Aludel’s existing eval stack to test and compare them, and then export the validated workflow as Jido/Elixir code.
There’s also a related feature already on the roadmap: letting people connect an executor so they can plug in their application’s real workflow/runtime and evaluate it through Aludel, instead of only testing inside the app itself.
So instead of Aludel only being the place where workflows get evaluated, it could become the place where they’re designed, pressure-tested, connected to real execution, and then turned into production code.
Curious if that feels compelling, especially for people already building with Jido.
V1 would be much simpler: just a visual workflow/diagramming tool on top of Aludel’s eval layer. Later, if the model holds up, Jido would be a strong target for turning those diagrams into executable workflows, instead of keeping them as static designs.
Don’t hold you breath yet as it’s just a decision point and there’s a lot of work to be done 
New release is up Release v0.2.0 · ccarvalho-eng/aludel · GitHub
Added
req_llm to 1.10.0
Since v0.2.0, Aludel has picked up a few meaningful improvements:
Callback execution mode is on main GitHub - ccarvalho-eng/aludel: LLM Evaluation for Phoenix Apps · GitHub
Now you can run your AI app’s real production workflow inside Aludel, instead of calling
providers directly from the library.
In :callback mode, your app provides an Aludel.Executor implementation and Aludel calls back into it for runs and eval suites. Aludel still handles prompt/version context, inputs, assertions, persistence, history, exports, and provider-based comparisons, while your app stays in control of the actual execution path.
This is useful if your app already has its own orchestration, retrieval, tool use, guardrails, or response pipeline and you want to evaluate that real path instead of a simplified direct LLM call.
Quick update: Aludel 0.3.0 is now published.
Hex: aludel | Hex
Docs: aludel v0.3.0 — Documentation
Release notes: Release v0.3.0 · ccarvalho-eng/aludel · GitHub
Highlights:
Install/update with:
{:aludel, "~> 0.3.0"}
Feedback welcome, especially from folks trying the embedded callback mode in real Phoenix apps.
I noticed this library has 140K+ downloads. I am bit surprised I’ve got no bug reports, feedbacks or pull requests. Please, let me know how useful this has been and if you’ve got any feature request or feedback.