Yes, you’re right. Different use cases here! But both valid.
Jskalc
bradley
Interesting idea. At first I misunderstood the library, but I think I grok it now. From what I can tell, the main overlap between our approaches is the eval syntax and the runner, but you’re leaning more into a custom YAML format. I’m not sure the YAML structure you’re using would be compatible with ex_eval, and maybe it shouldn’t be, since the goals of the two projects are pretty different. Does that assumption seem right to you?
That said, I’d be really interested in exploring some shared patterns between the two. I just pushed an update to ex_eval to support datasets as adapters, so in theory we could plug in various sources like YAML files, Ash resources, Ecto schemas, and so on. Not sure how well that aligns with your setup, but it’s something we could look into.
Makes sense. Before pushing ex_eval, I did the same in my early experiments and still use that pattern frequently for integration tests.
I love the idea of caching responses to disk with a custom Req step during tests. I’ll definitely consider implementing something similar to help manage costs.
I completely agree that pass/fail isn’t enough. I really like the dashboard you’ve built and the ability to drill into specific evals. What I’ll need eventually is the ability to track evaluations across different metrics, like pass/fail, scoring, min/max, and more. That aligns with what some of the bigger eval frameworks support in other ecosystems.
I also agree that having evals integrated with the codebase is a huge benefit. I don’t want to depend on an external system that I have to force into place.
This is actually one of the main reasons I’m looking to move away from ExUnit. It’s optimized for traditional code, but testing AI feels like a different paradigm. It might make sense to start fresh. I could be wrong, of course, and it probably depends on the use case. For me, I see evals as a form of code for the LLM, so I want an interface that makes it easy to scale and maintain.
Nice! I don’t see my non-technical teammates ever opening Cursor, haha.
All good. I think it really comes down to the use case and context. I’d love to keep collaborating and sharing ideas as this space evolves.
1
zachdaniel
Creator of Ash
In this case it’s not even meant to be used as a library. I want to make it a central tool we can run to evaluate LLM assistants skill with Elixir and its package ecosystem.
2
bradley
Just want to update everyone, I’ve been working on ex_eval in the background and think I found a pattern that I like. In summary:
-
Async-first with OTP supervision: Built on GenServer processes with proper supervision trees. You can run evaluations asynchronously and get real-time progress updates.
-
LLM-as-judge pattern: Instead of rigid assertion-based testing, evaluations use natural language criteria judged by LLMs. This is implemented using an adapter approach, starting with langchain as the base layer. (See ex_eval_langchain for the adapter if you’re curious)
-
Dataset protocol for flexible data sources: Whether your evaluation data comes from CSV files, databases, or inline definitions, the Dataset protocol provides a consistent interface.
-
Pipeline processing: Built-in support for preprocessors, postprocessors, and middleware. This lets you transform data, handle multi-turn conversations, and add custom evaluation logic without modifying the core framework.
-
Real-time monitoring: Integration with real time messaging for broadcasting evaluation progress, which is helpful for long-running evaluation suites and UI interfaces.
In terms of next steps, from here, I think I’m at the point where I can build a UI interface that is agnostic to the data layer. That way I can move on to implementing an Ash resource layer and then maybe other folks who don’t use Ash could implement Ecto for example.
Here is an example script (link) that I’ve landed on that’s inspired by Req and MLFlow:
Mix.install([
{:ex_eval, path: "./", override: true},
{:ex_eval_langchain, path: "../ex_eval_langchain"}
])
dataset = [
%{
input: "What is the capital of France?",
judge_prompt: "The answer should be Paris",
category: :geography
}
]
response_fn = fn
"What is the capital of France?" ->
"Paris"
_ ->
"I don't know"
end
ExEval.new()
|> ExEval.put_judge(ExEval.Langchain, model: "gpt-4.1-mini")
|> ExEval.put_dataset(dataset)
|> ExEval.put_response_fn(response_fn)
|> ExEval.put_experiment(:langchain)
|> ExEval.run(async: false)
Because everything is implemented as composible functions, we could in theory add whatever macros/syntactic sugar we want to streamline the overall DX.
Also, this project is really a proposal of sorts. I really only plan to add it to hex if people want it.
@zachdaniel just curious, are there any issues you think I’ll run into on the ash side implementing the data layer portion of this? Any considerations I should take into account?
0
zachdaniel
Creator of Ash
zachdaniel
Creator of Ash
bradley
zachdaniel
Creator of Ash
Sorry I realized maybe that wasn’t clear but I mean I’d want to see deterministic assertions in addition to an LLM judge.
I’m thinking a prebuilt frontend with a pluggable backend.
I’d have to know more about the goals of it on that front, but pluggable backend often means some kind of strictly defined API layer at which point other adapters could be provided perhaps, but if you’re in charge of both the front end and backend it sounds like perhaps complexity that isn’t warranted?
0
bradley
No worries, I know just what you mean. It’s a common pattern for eval frameworks to support different evaluation methods/metrics outside of LLM as a judge.
For the “pluggable backend” piece, I’m actually hoping the community can land on something foundational and reusable, think how Ecto is used for persistence, but Ash and others build on top. I’d prefer this evaluation code isn’t tied to a single framework like Ash, but rather lives as a shared library anyone can leverage (and extend with adapters if needed).
To be honest, my goal is a bit selfish: I don’t want to build and maintain this myself, but I haven’t found anything that fits my requirements yet. So, if I have to, I’ll put together something minimal to get by. David from Thinking Elixir just mentioned a similar idea in the latest episode, would love for him to chime in here if he’s interested!
0
zachdaniel
Creator of Ash
A pluggable backend sounds a bit strange though because you’d have a proscribed schema/API required. Do you mean like allow swapping a storage layer as opposed to the entire backend?
0