I’ll tell you what my real experience is, using LLMs has allowed me to write more well tested code, as well as more well documented code. For example, we’ve been having discussions about some issues w/ Ash’s documentation.

1. the search results didn’t seem ordered by the most helpful stuff
2. there is so much content that “cross referencing” is important, i.e putting links to things you were probably looking for if you got here and its not what you want.

I spent about 2 hours w/ Claude working on our four most commonly used modules, Ash, Ash.Changeset, Ash.Query and Ash.ActionInput. We have a solid usage rules file and other things that helped guide its output. I tasked it with expanding the docs on the most common functions (and many of the less common ones) ensuring that we have example usage, cross links to other relevant guides, and typespecs. It wasn’t “vibe coding”, i.e I reviewed all of the work, made many tweaks of my own, etc. But it turned a multi-day project into a 2-3 hour project. And when we have other docs issues in the future I will likely feed it into Claude and have it generate some nicely formatted output. i.e “this user had trouble with X, please look at the current docs for Y and produce something that would have prevented this confusion”. It also figured out why the search results were suboptimal before I could figure it out myself. The order of modules in the sidebar controls to some degree the order of modules in the search. We reordered the sidebar, and search got that much better

If you’re familiar with the issues around Ash docs, the improvement to this module is effectively night and day: Ash — ash v3.29.3

To me, it’s not a replacement for my brain, it’s just a way for me to exert more leverage than I would have before. When you know how to use it, it can make you faster, help you automate the boring stuff so you can focus on the important stuff, and get you past the blank-page problem.

Having spoken to Zach at ElixirConf EU and heard some of the challenges he has seen with supporting users that have completely fictional code. Coupled with his general skepticism of the AI hype. I am glad he wrote something up.

This is a pragmatic take on what we can usefully do around adoption, and being on a slimmer end of the bell curve when people start using bell-curve-machines for information and code generation.

Without trying to go to the moon or saying the cost of programming will go to zero. That stuff is speculative. This is mostly addressing current reality, whether you like the situation or not. I don’t love dealing with LLMs but I don’t particularly want to close my eyes because I don’t think it will entirely blow past.

treat them as if they are only capable of doing things like summarization and pattern transformation

They are not even capable of this, only of giving the appearance of being so. It’s on the prompter to check the accuracy.

I think the biggest direct risk for most users of LLMs are that the LLM companies are aiming to become gatekeepers. All these efforts to integrate with LLMs are free work for these companies, who will increase the rents once they get enough people dependent on them.