Keep product knowledge close to the code

I can fully relate to this one.

The project I’m working on right now includes complete documentation, plus the thought and backlog workflows in the repo. Being heavily AI-assisted, I found this a good way to make sure that the LLM has access to thoughts/backlogs as well as the “design” and “thoughts” which are not directly code but still shape the way the product behaves and should develop.

I have to admit that, in this sense, I no longer differentiate between code and documentation - both help the LLM understand the lay of the land. E.g. I have several doc gates as well - the simplest one being public function documentation being at 100%. Another “simple” thing I try to enforce is pushing the LLM to document why something is implemented the way it is, rather than how.

I have to admit that, in this sense, I no longer differentiate between code and documentation

Another “simple” thing I try to enforce is pushing the LLM to document why something is implemented the way it is, rather than how.

Yes exactly, couldn’t agree more!

Hi, I totally agree. Related to documentation, in docs we always put the how. But why this code exists is also important question!

I didn’t necessarily mean why it exists. More like why it’s implemented this way and not another. Things that are hard to deduce just by looking at how the code works.

Maybe I am misunderstanding the article and the comments here but I have found LLM-generated code comments to be a huge liability. In a team of just a few devs which use LLMs every day, heavily, they get outdated literally three days later in the best-case scenario.

What is being described in the article seems like a fragmented product spec to me, put right inside the code. It’s an interesting idea but I am guessing that it highly depends on the LLM you are using on whether it reads that or tries to find docs another place first.

Have you had improved success and less LLM turns the way you are proposing in the article, @ryanzidago?

Hey @dimitarvp,

In the article (and professionnally) I do not use of any code comments, I use doctests and moduledoc.

I use doctests to document a case what I do not want to handle right now, the doctest itself will fail loudly as soon as the edge case is fixed (which is the behaviour that I want).

So I guess I could return the question back to you, what do you suggest to use when we do not want to handle an edge case? How can we communicate these decisions to LLMs beyond the sessions during which we’ve made these decisions?

1. Writing a code comment? As you pointed out, those will become stale very quickly and fail silently
2. Updating the code to recognise the edge case and maybe return an error? Now you are diving into implementation territory, you need to make decisions (should I error, return nil, raise, etc ..) it’s another opportunity for the LLM to get something wrong; or maybe you don’t even want to update the function’s public API for fear of having to do many changes across the codebase …
3. A separate product specification / document that must be updated everytime the underlying code changes?
4. something else?

To me doctests seems the best compromise for that (they allow me to use prose to explains stuff AND I can encode assertions that fail loudly when they aren’t true anymore); though I’m very happy to hear from how other teams are handling those cases!

I use moduledoc to document the high level product and business processes that are less likely to go stale any time soon. Look at how the moduledoc does not contain:

• any API
• any structs
• any function calls
• etc ..

Of course you could have a separate .md file, a separate skill, but to me, putting those high level product processes near where they matter most seems also the best compromise.

I have to be honest too, I do not have empirical evidence or benchmark that this drastically improves LLMs ability to recall the correct rules when doing something. I only have a preference, an educated guess / gut feeling.

Doctests are indeed very helpful.

WRT comments going stale: Yes, that’s a real risk, and doctests do help mitigate it. I’ve also had very good experiences with doc gates—for example, holding documentation to the same standard as the code itself. Incorrect comments should be treated as build failures, just like broken code. In my case, though, this means I spend quite a lot of tokens on each gate run to make sure the docs are still correct. But I think the risk of documentation drifting is too high to ignore, since it can quickly undermine LLMs’ ability to add new code correctly.

Thanks for the clarification.

Moduledocs are kind of code comments anyway IMO but I do get your point: they are a better target. Random code comments are indeed much easier to go stale. Claude Fable is doing a bang-up job of auto-updating moduledocs for me, but that might be because me & it spent a lot of time together and probably at one point I told it “always maintain moduledocs” and it just started doing it. Not sure at this point.

Doctests are indeed a good way and they are something I plan to use more going forward, supposedly, however most of our tests are integration-level and the unit tests are also complemented by property tests so still not sure how I’d slot them in.

Again, thank you. Good ideas to think about.

Full agreement, it’s just that in my observation even an excellent model like Claude Fable still forgets to do it and I am not sure I want to spend as much time to craft more skills, more prompts, or even tooling (though Fable is very capable). Me and my team moved all documentation to a bunch of topical Markdown files and Fable is doing very well at updating them when the code changes; I barely had to do anything more than just tell it to remember to maintain them.