Code quality is code quality. Doesn’t matter who or what wrote it.

Claude gave me this today :

flagged? = opts[ :flagged] && something_else?( args)

..

// flagged? passed to a function it made it match against true or false

Fair enough. So maybe the metric is really wider, and it should be how to measure code quality period.

Either way I’m looking for good metrics for the quality of generated AI code from a concise prompt in particular as such a tool would likely be helpful improving use and or using less expensive models. There are many metrics to go by of course, but for me I think the following is what I look from Claude:

• Code that fully meets formal specifications and requirements.
• Flexible code architecture that is easy to change, extend and test.
• No major performance issues.
• Secure handling of authentications and authorizations and other common security topics.
• Robust test coverage.
• Good documentation.
• Idiomatic best-practices code as per the language.

Getting to a level where Claude can be trusted to reliably and easily do that is a worthy I goal I think.

Cool and relevant:

https://github.com/Cinderella-Man/credence

I have zero idea but – really good question.

I’d start with “less coding lines”. LLMs are master bullsh1tters and will spit out 1000 lines of code when I, given the time, can very likely write 250 and even have the code be more understandable and nicer to work with.

I like it. Have to check the degree of overlap to what is in place already, but certainly relevant.

That is a good point. Short and straight forward readable sounds good. (In my mind longer than needed would be bad, and shorter and too intricate/ clever/ compressed would also be bad). If we assume the long version actually boils down to about the same functionality in the AST or compiled code, then it should be possible to reengineer the shortest clear readable source version from that. Anything longer could be deemed wasterful. (Thinking loud, maybe something sticks).

Well let’s be honest here. If you achieve that you can command $20k / hour to rework the entire world’s code into something better. Or maybe $300k / hour.

But yes, this is the ultimate dream. Maybe you and @mudasobwa should talk: he started his project for a more-or-less universal AST with the idea to detect stuff beyond what credo and various other linters can detect.

That would certainly justity buying some Threadrippers.

Given some more thought the compiled to reengineered source seems too high level for me at least. But reengineering from the AST back into short source should be possible for a module or boundary with finite defined inputs and known outputs. Basically turning it into a perfect black box which can be optimized in every which way as long as it upholds the fully testable the input-output relation. I’m not sure what percentage of code that would be applicable for, but for a functional language that should be a fair bit.

I do have this test for ‘should this just be a lookup table’ instead which basically just do the same thing for simple functions.

This might be too obvious and I apologize if that’s the case: using one agent to write the code, another to test the code, and a third to review the code has paid large dividends (at least with gemini). It’s circular, using the AI to review the AI’s work but doing so with a new context-free agent seems to catch issues ranging from “doesn’t match the spec” to “this function is too long.” I tend to use a lot of agents in the course of a task. Gemini seems pretty reliable at generating “hand off” instructions.

I’ve also found that being super specific about my acceptance criteria helps both with development and review. “Good documentation” is not specific enough. I have something like “all public functions must have a doc, all modules must have a moduledoc, and all private functions must have a comment explaining what they do.” Much like with the specs I begin with, the more specific the constraints the better the AI does meeting them since – I assume – there’s objective criteria to measure against.

Nah, the code talks and I walk.

There is zero sense in having shorter code, if that is really needed, go rename all the functions to f1, f2, and all the variables to v1, v2, and squeeze all the spacing, and whatnot.

The actual size of code does not really matter at all, once we have MCP exposing smart tools and RAG built on top of that. (I went this path, btw, and I succeeded to some extent.) The code itself is out of the equation immediately after your MCP covers all the needs (requests) your LLM produces. Instead of grep and sed your own RAG can do better, that’s the most important thing here.

Code is something your LLM should not deal with at all.