150,000 Lines of Vibe Coded Elixir: The Good, The Bad, and The Ugly

150k LoC doesn’t mean anything if you don’t have a of declaration of what it does.

Our Elixir project is at 31k lines cloc --exclude-dir=node_modules,_build,deps . and I will remove a few thousand since this is in the middle of a refactor and I haven’t been cleaning up things thoroughly.

It’s a fairly simple app, a full admin, api for mobile clients, orchestration logic via Oban to process data, spin up servers etc. Some 3rd party integrations.

I use Tidewave quite a bit and I’ve noticed the Anthropic models (haven’t tried others) are happy to duplicate logic like crazy, this is why I’m wary about LoC as an indication of anything really (and personally I want to keep it low, not high).

Does it matter how many lines of code there are in nobody reads them? Does it matter if logic is duplicated if nobody ever has to change it?

Everyone trying to apply good software practice to vibecoding (“vibe engineering”, “subagents”, “orchestration”) is going to get wrecked by the bitter lesson as usual. Differentiability is king.

Either retreat upmarket or embrace the vibes.

Someone or something will read them and LLMs handle duplication worse than humans in my experience.

And then assuming we should just sit idle when our model provider of choice have downtime seems like a risky proposition.

As for good practice in general, the code runs on computers and those haven’t changed, good code runs cheaper and faster.

Curious what upmarket means to you. I could see it as higher quality, better designed software from a UX standpoint as Maggie alludes — the underlying code may or may not be “better” — or software that can’t be done by LLMs which I suppose might only include building the models themselves.

We did find while developing it liked to duplicate code

I touched on this a little bit in the “AI Can’t Organize” section. It’ll happily rewrite code it thinks it needs if it doesn’t know where it exists, or where to look for it

so it’s important to keep codebase architecture consistent, but have some form of documentation the AI can read so it can find logic that already exists

It’s a little bit like onboarding a junior dev with amnesia every day

It means expensive enough to justify the up-front cost of human development when models are orders of magnitude cheaper. The quality is downstream of that because nobody will pay for junk when they can get junk for free.

If you want to keep programming, go where the models cannot. Models won’t be able to replicate a level of quality outside of their training set for a while, and while it’s easy to RL tests that pass it is very hard to RL UX. So yes upmarket means quality, but it’s not a meaningless heuristic; you have to understand why the models can’t get there.

I have followed “AI” closely for a long time and I always believed this was coming, so I am perhaps not as shocked by the progress as some. To be honest, what did surprise me was just how hard it is to hit the level of quality I believe is needed to succeed in the new market. This was partly a skill issue (I have gotten better!), but I was also spending a lot of time swimming upstream with tooling.

You are aware of my solution there

This is wrong, for a subtle reason. Bad code leads to bugs, and bugs destroy UX because they harm user confidence in the software.

All good if the software is cheap, but nobody will pay for junk.

This is a bit surprising to read. Opus 4.5 helped me do a comprehensive refactor of my app’s test suite, and concurrency was a big part of that. It did deep dives into each domain and identified which tests are safe to run concurrently, and we were able to make the suite run 40% faster overall. It also fixed several intermittent test failures caused by concurrency, as well as the dreaded DBConnection errors that tend to clutter the test outputs.

Yes. I use supertester | Hex for building robust concurrent tests. Your prompt might say something like, “Add supertester latest version from hex to mix.exs and get deps. Use supertester to refactor the entire test suite per these docs: <ctx>{docs text}</ctx>." with the content of these three documents as the {docs text}:

https://github.com/nshkrdotcom/supertester/blob/master/README.md

https://github.com/nshkrdotcom/supertester/blob/master/guides/MANUAL.md

https://github.com/nshkrdotcom/supertester/blob/master/guides/API_GUIDE.md

The library arose from the need to address the same issues experienced by @John-BoothIQ. Hope this helps someone. I use it on many of my projects, and it works well.

This is both encouraging and disappointing.

Encouraging because those DBConnection Errors is exactly what I was hoping AI would be great at.

Disappointing since I couldn’t get it to do the same

Do you remember what you told the model to try and track those down?

Does it matter how many lines of code there are in nobody reads them? Does it matter if logic is duplicated if nobody ever has to change it?

Duplicated logic, at least when it amounts to duplicated knowledge, is a change for inconsistencies, which means bugs. Simple example: the software lets you create a username with format X, but then it won’t let you log in unless it’s format Y.

Also, in my experience, LLMs perform way better on small codebases (or targeted sections of them) than when they must read a large amount of code. And they do way better at “add another component/test that follows the pattern of existing ones” than at starting from scratch.

So if a codebase becomes a giant pile of duplicated mess, then yes, I suspect it matters a lot for how well LLMs can continue to work on it, just like it does for humans.

Quality will always matter.