I think you’re correct that there’s a lot of great wisdom about the right way to build apps on the BEAM and in Elixir. The problem for us is that we’re conflating several different concerns and trying to shoehorn all of them into “code quality.”
If we’re directly measuring the output of large language models, it needs to fall into several different categories so we can break down this problem and have a fighting chance of addressing each one. The first and most important thing we need to measure is correctness. When I say correctness, I mean at the application level, not the module level. You can have a collection of correct modules that do not compose into a correct application.
The second most important thing, in my opinion, is architecture. Did the model use the right tool for the job? Did it put the code in the right place? Did it respect the architecture decisions that have been made about the application? The third is security. It’s important enough that it deserves its own category. And the last and least important, in my opinion, is actually code quality.
There are potential additional buckets worth considering - performance, maintainability, observability and instrumentation, documentation. I’d argue most of these fold into the categories above.
The reason I’m proposing this breakdown is that it makes each problem easier to solve, and it helps us better understand and communicate about the solutions this community is already presenting. Archdo obviously fits into architecture. Parts of living substrate fit into architecture. Parts of the cleanup guide fit into architecture. Ragex has security rules. The cleanup guide has security rules.
The thing I find most interesting is that everyone on this thread is more focused on code quality, security, and architectural compliance, and less focused on correctness. I’ve done several harness builds and experiments, and I’ve achieved a great deal of success generating full Elixir applications by focusing myopically on correctness.
Following up on what I said earlier about the bitter lesson: after sitting with it, I don’t think this approach conflicts. We’re not trying to encode human knowledge into the model’s decision making. We’re offering procedural resources to the model to use, because static checks are just more effective. We don’t want to roll the dice with verification.
So my question back: does it make sense to try to break this measurement problem down into these characteristic components, so each one can be measured with the tools and discipline that fit it best?
