Thanks for the insight, I can imagine how huge healthcare is, not sure how one would navigate around the hipaa land mines. Insurance feels elusive to me so I will need to do more research on that.
Tech support sounds like a solved avenue. but anyway you got me interested to look into RAGs again & your project
Don’t overestimate how much has been solved.
If you think tech support is anything like a solved avenue, you’re welcome to call your ISP to troubleshoot your modem. If it were solved, I don’t think it would be that much of a headache! 
I did have a recent calling with my bank and it was a headache, I just meant its probably harder for a startup to disrupt that space and sell a package, since theyve been trying to automate stuff before llms were a thing vs like a eye doctor or something
Continuing!
First: I want to be forthcoming about my use of AI. I wrote the prose in this post by hand. However, I did use AI to find the code snippets and property tests for citation provenance in this code base. I used prompts like
Find the property tests in this project relating to citation provenance and explain them with code snippets drawn from the actual files. I want filenames and line numbers both for the relevant tests and the code under test.
and
…please show where [TDD commit discipline is] stipulated in the agent contract and some examples in the commit history with code snippets and commit hashes. Put it in markdown format, please.
The code snippets below were copied and pasted from Claude Code’s output. So was the snippet from the agent contract, which, itself, is AI-authored.
Now for the fun part!
I want to talk about testing. This is a good opportunity to examine the impact of agentic coding on multiple levels. In particular: agent contract, testing philosophy, and the intersection of those two in the development workflow.
In the agent contract, I enforce TDD discipline. That means my commits, where possible, come in pairs. The first commit is generally a red test commit. I require the agent to write a failing test and commit that before writing any app code. This stone kills several birds. First, it circumvents the tendency of agents to write slop and then write tests that the slop passes, because you’re enforcing TDD. The tests must fail first, and this increases the probability that they’ll be nontrivial. Incidentally, on of the Credo checks I stole from Mike Zornek checks for weak tests, and it has already caught a few of those at the red commit stage.
## Testing: Ordering and Failure Handling
**Ordering — for any behavior change:**
1. **Spec.** User describes the change.
2. **Tests first.** Encode the new contract in tests before touching implementation. If the change is non-trivial, push the test diff for review and STOP for approval.
3. **Human reviews tests** — the tests are the spec.
4. **Implement** against the reviewed tests.
5. **Run gates** (Quality Gates above). Iterate on the *implementation*, not the tests, until green.
6. **Stop and ask** if step 5 keeps failing in ways that suggest the test itself is wrong.`
Tests written after implementation encode what the code did, not what it should do. Write them first.
You’ll notice that this agent contract does not, specifically, stipulate red commits. It does, however, require the agent to “push the test diff”, which, in practice, requires a commit. So the red commits are, strictly speaking, an implementation of the agent contract.
In terms of testing philosophy, I lean heavily on property-based tests. If you’re not familiar with property testing, I’ll give you a quick refresher: a property-based test is a test that enforces an invariant. It does this by generating hundreds of randomized inputs, and then checking that the outputs always follow a defined rule. For any logic with a nontrivial range of possible inputs – defining “nontrivial” here as inputs that could surprise you – property tests are plausibly better at finding corner cases than ordinary test cases.
For example, in my RAG project…
RAG works by looking for documents in a database and using them as context to prompt an LLM. We want to be able to cite the things we look up. For example, if a customer asks an insurance company bot about a policy, our RAG program should be able to look up the policy document, generate an answer based thereon, and cite the actual document with title and page number or some other set of identifying info. It cites these as numbers inside of brackets, Wikipedia-style: This policy stipulates that such-and-such [3], where [3] links to the cited resource. The system prompt for this runs,
Use inline citations like [1], [2] when drawing from a specific chunk. … If multiple chunks support a claim, cite all of them like [1][3].
Because citations are in the text generated by the LLM, there is a chance they can be hallucinated. We want to catch any hallucinations that occur. We do that with Citations.extract/2. That function looks every citation and checks to see if it was hallucinated. For each [N] that is deemed hallucinated, it logs a warning and also returns the hallucinations separately. Note that a hallucinated citation here is one that was made up by the model; we do not, at this juncture, check to ensure that the cited material really supports the LLM’s output. Right now, it’s up to humans to verify that.
But how do we know that Citations.extract/2 actually does what it’s supposed to do? What if it accidentally labels a real citation as a hallucination? Well, we can test it using property testing! We property-test to ensure that extract/2 never mislabels a real citation as hallucinated. The property, “hallucinated indices are disjoint from chunk_map keys”, stipulates that no citation deemed invalid is ever inside the chunk_map that contains the actual chunks being cited (which come from our document corpus not an LLM).
The Elixir code implementing this property is:
# test/cake/citations_property_test.exs:69-77
property "hallucinated indices are disjoint from chunk_map keys" do
check all(
map <- chunk_map(),
text <- text_with_markers(Enum.to_list(1..32))
) do
{_valid, hallucinated} = Citations.extract(text, map)
assert Enum.all?(hallucinated, fn idx -> not Map.has_key?(map, idx) end)
end
end
chunk_map/0 generates a map taking random positive integers between 1 and 32 into metadata maps. There are (at most) eight keys in that map, each of which is one of those integers. Meanwhile, text_with_markers/1 builds a text string made up of segments. Each segment can be a random alphanumeric of length <= 8 or random integers from the index_pool formatted as strings thus: "[3]".
Because the two random things (the chunk map and the text with markers) are generated separately, they have no relationship to one another and can match or mismatch. They both come from a range of 1..32, but they’re both random and independently generated. So some will match and some will not.
The actual property is defined on line 75:
assert Enum.all?(hallucinated, fn idx -> not Map.has_key?(map, idx) end)
We are asserting that, for any value in hallucinated, there is no key in the map of citable chunks corresponding to that value. If you ever studied philosophy and remember your first-order predicate logic:
∀x: Hx → ¬(x ∈ C)
Where Hx means “x is hallucinated” and C is the set of citable chunks. We can also express the disjoint nature of the set of hallucinations and cited chunks thus:
(H ∩ C) = ∅
Strictly speaking, this is not formally proven. It is, rather, tested for a set of 100 random inputs. The power of property testing is that those 100 random inputs are likely to cover corner cases that I wouldn’t have thought of while writing the test because I wasn’t thinking of them when I wrote the code.
One more thing. There’s a trivial solution to this property:
def extract(_text, _map), do: {[], []}
But there are other properties in the test file that close this gap.
Back to the macro: how does this change our development workflow? Well, in priv/reference/creating-things.md, we have:
Always favor property tests where possible: while reasoning about testability, consider what properties each function ought to have and whether a property test can cover them; if so, prefer property tests over ordinary tests.
It’s very simple. We have three things that combine in the dev workflow:
The upshot of all this: the agent begins by writing a red failing test commit. It runs the test and ensures that those tests fail. The tests in question are very often property tests due to the specification above. Therefore, when the agent writes its green commit, it cannot overfit to the property tests just made in the red commit. The enforced TDD discipline that the agents use here thus has cash value because it forces failing tests and rigorous adherence to invariants that we cannot cheat by overfitting.
(Fun fact: while working on this post, I realized that creating-things.md is loaded for a much narrower class of problems than I would like, so property testing has perhaps not been as strong as it could be. This prompted me to make a change to the agent contract, which Claude Code is working on as I write this post. 
)
One last thing: I should give you an example of overfitting. A case of overfitting would be if I had the agent write a traditional test, hardcode a list including hallucinated citations, and then show that Citations.extract/2 returns another hardcoded list. That’s the sort of test an agent might write if you both fail to prefer property tests and allow the agent to write the tests after the code. This whole edifice of property testing and TDD discipline in the commit workflow is meant to forfend that class of issue.
Oh, definitely. The problem is that big orgs do these things internally (frequently with sub-par solutions) so your proving ground for this kind of thing will be small-to-mid-sized local firm. There are places like that that do things like keeping all their knowledge on a network drive. Their knowledge management system is literally a person who knows where all the directories in the network drive are.
How would you go about engineering something like this? I looked into local stt → llm → tts about a month ago, for a small project and the economics didnt look to great and I ended up on the path of gemini live, pricing was okay and the promises were great, ofc google has unlimited compute, but realistically you still want to own the whole stack especially for working on projects that demand privacy & plugging in your own models + data. But now the more I think about it, if i had a dedicated gpu and a couple of small models running , all I would need to do is have a really good batching architecture & most users would get responses quickly and the cost wouldnt be astronomical and I wouldnt have to use gemini live and elixir is great for things like this
I haven’t built anything like that yet, but I can tell you how I’d reason about it…
In the first place, STT/TTS is a whole lot less hungry than an LLM. So if you’re running a local LLM, I’d give the GPU entirely to the LLM and have the other stuff on the CPU. That changes how we do batching. You can run Whisper at fairly acceptable speeds on a consumer dev laptop without doing any fancy GPU stuff (I learned this while futzing around with Bumblebee).
Since, by hypothesis, Whisper is CPU-bound, our batching architecture is gonna look like parallelization of multiple Whisper instances across CPU cores. We have a layer that collates user utterances into batches and feeds those batches into Whisper instances. The resulting text goes into a queue of some sort to be served to the LLM, and the LLM’s output needs to be sorted and batched again for TTS, which is gonna be parallelized over cores similarly to STT. Again, by hypothesis, it’s CPU-bound.
TTS wants to stream rather than batch, of course. So the strategy for taking the LLM’s output into TTS instances needs to accumulate in such a way that we don’t get any hiccups in the middle. That’s a natural tension in this whole thing, because STT/LLMs like batches and TTS likes streams.
So in other words: you don’t have “a” batching architecture. You have 3: it goes users → STT → LLM → TTS and finally fanned back out to users, and each of those arrows is its own batching architecture.