Vidar
How to measure AI code quality?
I’ve been trying to come up with a fairly real life representative way of evaluating code quality from AI, and by extension the functionality of skills and their setup, and the code checking utilities used.
Ideally there would be perfect real life scope code examples along with the prompts that should make them. And a variety for different problem spaces to make sure it is widely applicable. One could then setup a loop using the ideal prompts where
- The planning and implementing skills act according to the prompt.
- The review tools and review skill telling the plan and implementing skills what needs improvement in order to get closer to the code ideal.
- If/ when the planning and implementation skills get closer than the review tools can actively instruct then the review tools and skill should improve. Given the non-deterministic somewhat noisy nature of LLMs this cycle should gradually evolve towards the better.
Alas, for lack of real life scale ideal code examples with ideal prompts the best I have found so far is the Real Life Conduit which is a medium clone setup. It is just one data point, but it will give a certain objective feedback as it is fully external, made for testing purposes, and can tested online against their live API.
So that is the current plan. Give a short prompt without much technical guidance, and let Claude sort it out or fail horribly. To what extent will be some kind of metric I suppose? So I’m thinking: 1. Going blind 1 shot from the prompt and directly to test against the API. 2. Going blind 1 shot, but before testing against the API use code review tools and the review skills and have Claude ‘use tools and review skill and fix any and all issues’. And then test.
I think it is also interesting to repeat that with Claude on lower thinking modes. Given enough skills guidance, and or with automated code check and fix after, maybe a lower level thinking or lower level model will be make quality output? In that case good skills and review tools would be cost and token saving, which I would appreciate a lot.
Anyway, ideas for good functional metrics are welcome?
Most Liked
jdiago
Code quality is code quality. Doesn’t matter who or what wrote it.
dimitarvp
I have zero idea but – really good question.
I’d start with “less coding lines”. LLMs are master bullsh1tters and will spit out 1000 lines of code when I, given the time, can very likely write 250 and even have the code be more understandable and nicer to work with.
mudasobwa
Validating the quality of the code, produced by the very sophisticated code completion tool with another code completion tool always looked silly to me. That’s why I extensively developed static code analysis and linter tools in the last several month.
Additional credo checks helped me to catch stuff several times → OeditusCredo v0.8.1 — Documentation
Literally last week I caught and broke the huge compile-connected mega-cycle (459 nodes) with Ragex.
Old good static analysis works with the code on the level, LLM could never ever achieve, that’s why.
Popular in AI / LLMs
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"