Afaik this is due to the context window which may be too small to hold the whole code base ‘in memory’. Hence duplication: some functions are simply not seen.
AI workflows get better so a preprocessor can pick relevant parts of the code for the context and use this subset.
You tell me after the app is hacked and customer data leaked. Tell me and the responsible agencies.
If you care about security, correctness, quality, or craft I don’t think you should be letting these models anywhere near your codebase. I certainly won’t.
But realistically most of the industry stopped caring about those things long before LLMs, and realistically these models have probably already surpassed the security skillset of the average webdev anyway.
Implicit in this point, though, is that I expect those things to change. Security and correctness will come easily, probably faster than you think. Honestly probably this year. Models are already good at finding security bugs, so a little adversarial RL will go a long way.
Quality is a lot harder, in part because it’s a matter of where you put the goalposts. My bet is the models will be outputting mediocrity for a while. Because their training set is mediocre, and because it will be hard to RL UX as it’s essentially AI-complete. But we’ve been putting up with mediocre software for decades now, so honestly I don’t expect things to get dramatically worse. If anything it will free up the real programmers to actually spend their time making good things again. Glass half full.
And that gets into the last one. I’m not sure if the models can ever replace “craft”, because to me that’s in the eye of the artisan. And there are still artisans all over the world making things exactly as they have for hundreds of years, industrial revolution be damned. But do not let that analogy distract you from the fact that, unlike physical goods, computer programs can be infinitely copied. The playing field is much more level.
For us they were caused almost exclusively by async code spawned by liveviews. When the liveview request completed and we asserted on the results, the test process finished and exited before their async tasks (which we didn’t care about in those specific tests) could finish running. This caused existing database connections from those tasks to get prematurely killed.
I think we handled most of the errors by explicitly rendering the view using render_async. For example:
It adds a bit of verbosity but we don’t mind it.
That’s a good find! I’m not sure I’d have thought of that on my own. I’ll add that to my test investigator rule for the future
Thank you!
Great writeup. My experience overlaps but with a different mitigation strategy.
I’m building an FBP (Flow-Based Programming) framework in Elixir — visual pipeline editor, distributed processing, 30+ node types. Using Claude Code daily.
The “AI can’t organize” and “OTP/async is ugly” problems hit me hard early on. My solution: don’t let AI make those decisions at all.
I built MCP servers that enforce architectural constraints at the tool level. AI can’t violate patterns because the tooling prevents it, not because I correct it afterward.
For the OTP blindspot — the FBP node abstraction hides it entirely. Node authors implement handle_message/2, the framework handles GenServers, supervision, message routing. AI doesn’t need to understand process lifecycles because it never touches them.
Basically: constrain the problem space until AI’s weaknesses don’t matter.
The defensive/imperative code issue persists though. Still correcting if/else chains that should be pattern matches.
Curious to how you’re dealing with this. Have you tried proactively using CLAUDE.md/AGENTS.md to advise against this behavior or to encourage using pattern matching instead (where appropriate)? Specifically call it out during code audits? Something else?
Yes, CLAUDE.md has explicit style rules — prefer pattern matching, use case/with over nested if/else, let it crash instead of defensive nil-checking.
Then Credo periodically to catch what slips through, and warnings-as-errors in compilation so nothing sloppy gets past. Not bulletproof but catches most of it.
For context, I’m not running swarms of agents overnight. Single session, supervised, constrained by MCP tooling. The AI implements what I’ve designed, not the other way around.
Anecdotally, the model responds much better to the word “destructure” instead of “pattern match”.
For example, if I tell it to “pattern match in the function head”, it kinda flails.
If I say “destructure the arguments in the function head,” it responds much better.
I attribute this to JavaScript’s object destructuring, but that’s just a hunch.
