Using doctests to understand a feature

After writing about the importance of executable evidence and explaining how I’m using doctests to document known limitations and moduledoc to specify high level product processes, I’m now writing another article about documentation.

I often find myself asking LLMs to explain me a complex feature. As a result, they may return quite some verbose text which isn’t pleasant to read all day long and takes me time to parse. I find code easier to reason about than English in some cases anyway.

Sometimes, I even ask them to back their statements with evidence, such as tests, which helps them correct their false assumptions.

At some point, I realised that I was using docs and tests separately, i.e. as a weaker version of doctests!

I could actually just let the AI agent explain me the feature through a high level overview as a moduledoc that is roughly of the following shape:

## Section title
What (observable product behaviour)
Why (product behaviour rationale)
<concrete illustrative example with doctests>
...

The more I work with LLMs, the more I love doctests.
They make the specification much easier to verify.
You can reduce drift if you keep the doc part thin and focused on the why.

Such a powerful feature for both humans and agents alike!