Yes, the idea is that there is a core Store interface (link) that can be implemented by any storage layer. I went ahead and pushed ash_ex_eval for you to see. I one shotted it with claude code so it needs a lot of work but hoping it’ll demonstrate what I’m thinking.
bradley
0
zachdaniel
Creator of Ash
zachdaniel
Creator of Ash
If the goal is to allow people to bring their own database, then you’d want to look at allowing people to swap either the Ash data layer or the Ecto repo that powers some piece of generic code IMO, as opposed to having entirely different implementations of the same app.
0
bradley
The repo is entirely a mix task that generates all of the resources needed for evaluations. The code also generates a Store implementation (link) which then allows users to use the pipeline syntax above for doing evals.
ExEval.new()
|> ExEval.put_store(MyApp.Evaluations.Store)
|> ExEval.put_dataset(dataset)
|> ExEval.put_response_fn(response_fn)
|> ExEval.put_experiment(:my_experiment)
|> ExEval.run()
The intention is to be able to bring my own database resources and then modify as needed, kind of like your approach with mix ash_ai.gen.chat.
Later on, there’s no reason why some web layer couldn’t just implement the pipeline syntax above, along with some of the other things provided in ex_eval for realtime/liveviews.
0
zachdaniel
Creator of Ash
webofbits
I’ve been thinking about this problem from a slightly different angle and have been building Aludel
( GitHub - ccarvalho-eng/aludel: LLM Evaluation for Phoenix Apps · GitHub ), a small Phoenix-native workbench for prompt evaluation.
At work, we have some services in which we’ve built some prompt management/evaluation dashboards so I’ve thought it would be nice to have something more generic that teams could re-use instead of building a dashboard each time.
It’s less “an ExUnit-like eval DSL” and more “a place to compare runs, track prompt versions, and execute regression suites with visibility into outputs, latency, token usage, and cost.” So it’s probably closer to app-facing eval workflow than to a general-purpose framework.
Still early, but the thing I’ve found useful is keeping eval work close to the app and data model rather than pushing everything into external spreadsheets or one-off scripts.
I’d be very interested in how others are balancing:
- code-driven evals vs dataset-driven evals
- embedded-in-app workflows vs external tools
- reproducibility vs iteration speed
Thread here has been helpful for that.
0