The top model on Aider LLM leaderboard is claude-3.5-sonnet.

I have switched from OpenAI 4o and in my experience in Elixir claude 3.5 sonnet is way ahead.

I agree with you that LLMs absolutely give no guarantees that they’ll always be correct and that they can be confident in their failures. However, the lesson from an example like this shouldn’t be to conclude that LLMs aren’t useful or helpful in general. They are both useful and helpful more than they’re not… so long as you’re using them correctly and come to them with proper expectations. They are not going to write your application for you or replace the programmer: in fact, I think it takes a more senior developer to really extract the benefit from these tools at this point. I look at LLM given solutions/code much the same way I might approach a Stack Overflow answer to a question I have… often times I find at least enough to set me on the right path… but I also see answers that, through the lens of my experience, I know somehow just “aren’t right” or complete, etc. If I ask the LLM to write code for me, I’m now viewing it very much as a junior developer whose code I need to carefully review. The answers or code don’t always work out or save time, but over repeated iterations, I find that overall I’m clearly more productive.

I also find that these things can be helpful in just explaining the code written by others. What turned my opinion on LLM utility was a task I was asked to do. I was given an MS SQL Server T-SQL routine to convert to PostgreSQL PL/pgSQL which was a long mess of nested loops and conditionals. Another developer on the project asked an LLM to convert the procedure to PostgreSQL and it didn’t work so I got the ask. After my failing to get a correct result a couple of times (missing a conditionally set value a couple of nesting levels away kind of things and not having an example of what actual inputs or outputs were) I turned to the LLM… but I didn’t ask it to create the code, I simply asked it to tell me what the T-SQL routine did in plain English. The LLM got that description spot on, which I could verify, and I was able to produce a correct results (by hand) in a single well structured SQL query at around a third the amount of code. At that point I was sold but also understood the role the LLM should play in the workflow.

Finally, we mustn’t forget that humans can provide wrong answers with strident confidence as well. The original posing of the “Monty Hall Problem” (Monty Hall problem - Wikipedia), and many of subsequent responses to the published solution, demonstrated that, while we may fail in different ways than an LLM, we still can fail with great certitude that we’re right. So I will continue to ask for help from colleagues and delegate tasks to other humans, too… but I’ll also consider the possibility that the answers I get will be wrong or flawed in some way and so will never suspend my judgment or critical evaluations.

Auto commits were annoying for me too so I have this in my .aider.conf.yml:

  ## Enable/disable auto commit of LLM changes (default: True)
  auto-commits: false
  
  ## Attribute aider code changes in the git author name (default: True)
  attribute-author: false
  
  ## Attribute aider commits in the git committer name (default: True)
  attribute-committer: false