We are in the process of beta testing a feature more or less identical to this, although we needed to support fairly complex filters across multiple tables and in some cases with multiple params. We already had a conventional UI built out, and the backend to support the filter logic.

One thing we did was use the existing function and module docs from the filter logic using Code.fetch_docs to build the LLM context. And then we added a bunch more detail and examples to function docs until the LLM was able to reliably return useful data. The filter logic also already had casting/validation so we could just use that. Took some fiddling, but the entire feature only ended up being a thin wrapper around ReqLLM, a context generator that pulled in all the docs, and an endpoint that called out to those and applied the validation and returned the result. And as a bonus we had more general purpose docs than when we started. Turns out a lot of the instructions that help Claude also help human devs.

I maintain a small utility for dumping docs and as a next step I am going to try to use that to extract the “context baking” logic from runtime code.

Definitely a fun experiment! The challenge of validating the response sounds complicated though