Thinking Elixir 259 - Chris McCord on phoenix.new

[bradley]

Hey! David, what are you hinting at around AI evals!? I have a discussion going right now about this, trying to get more conversation going. Would love to hear what thoughts you have.

How is the community currently evaluating AI applications?

Hi everyone, I’m curious how people in the Elixir community are approaching evaluation frameworks for AI applications, whether you’re using Ash, Ash AI, or something else entirely. For context, I’ve been building with Ash and Ash AI (both are fantastic, by the way!) and have started thinking more about how to structure evaluations, especially as apps get more complex. In other ecosystems, there are tools like Ragas, MLflow, and LangSmith that help with LLM evals, red-teaming, RAG scoring, and …