I think this argument rests on a dichotomy that does not hold up to critical scrutiny. On the one side, you have institutional competence, which is rigorous and authoritative. On the other, you have LLMs, which are “statistical guesses” and “incompetent imitators.”
But what is institutional competence, if not consolidated statistical knowledge? Building codes come from observed failure patterns, updated after every collapse. Water treatment standards come from epidemiological data on outbreaks. Medical practice comes from clinical trials, which are literally statistical inference at industrial scale. The shoulders we stand on today are built from statistically-validated patterns, formalized into codes, reviewed by experts and enforced by liability. Distilling LLMs down to “statistical guesses” to contrast them with institutional knowledge treats institutional knowledge as something it isn’t. Because at the end of the day, it is statistics all the way down!
In reality, the difference isn’t the statistical nature of knowledge, but the verification layer around it. For us, this layer includes tests, type systems, linters, code review, CI pipelines, staging environments, error monitoring and post-incident review. These are our building codes. And if anything, the AI era has raised the bar on all of them. Test coverage is no longer optional. Behavioral validation requires significant expansion. Code review hasn’t disappeared, it shifted from “is this line correct” to “does this actually solve the problem”. These are the types of institutional responses to a new tool: absorb it, strengthen the verification layer around it, and end up more robust than before. This is precisely the opposite of erosion.
This actually matches your own scenario, if you look closely. You ask: what would happen if the water company replaced purification expertise with an LLM? But no water treatment engineer is doing that, are they? Professional LLM use in engineering involves drafting, exploring, checking against codes and calculations, routing through existing review and liability structure. The LLM is an input into the verification layer, not a replacement for it. Your scenario describes incompetence using LLMs, not LLMs causing incompetence. And that failure mode predates LLMs.
Ultimately, what you are saying is a new version of the pocket calculator argument. And the encyclopedia argument. And the Wikipedia argument. And the Stack Overflow argument. Each new tool, each new layer of abstraction, got the same treatment: people said it would erode institutional competence by letting practitioners bypass real understanding. In each case, the critics were partially correct about skill atrophy (it is inarguable that we lose skills that we don’t use frequently), but decisively wrong about the net institutional effect. Bridges today are more stable than ever, medicine more effective, drinking water much cleaner, and airplanes dramatically safer. The institutional knowledge didn’t erode at all. Rather, it consolidated into codes, review systems and testing/verification regimes on top of the new layer. It became more durable.
There is a real failure mode in the neighborhood of what you’re describing, which is practitioners using LLMs without a verification layer and treating output as authoritative. This is worth pointing out and worth guarding against. But the remedy is maintaining and strengthening the verification layer, which competent institutions have always done for every new tool and knowledge source.
(fully anachronic from the above story, found in a flea market last year)
