Ever improving LLM models, still have to copy and paste documentation from hexdocs. Can we do better?

Roughly three better solutions:

• You can create an MCP server akin to GitHub - 9elements/hex-mcp: MCP Server for Hex Package Versions · GitHub
• Use this to convert hex documentation to .txt: GitHub - mjrusso/hex2txt: hex.pm package → llms.txt · GitHub, cf. also https://hex2txt.fly.dev/
• Wait until this lands natively (for newly generated docs at least): https://github.com/elixir-lang/ex_doc/pull/1976

A combination might even yield better results

I’m very curious about the actual solution you came up with (tangential to your topic I know). Always looking for a way to improve image. And background removal is something I’ve been investigating.

Don’t all of the big providers have search tools now? I think a general solution is always going to win here.

Honestly I always have the problem of finding the correct documentation, especially when it comes to bigger things such a phoenix.

I don’t think that specialized prompts for a bloated LLM is the way to go, as there is no information where the data was scraped from, this is especially a problem since a library usually has multiple versions, including major versions that change or introduce new things.

I think that fine-tuning a model such as BERT with elixir documentation might be something with a lot of potential, especially since we can make it aware of different versions of libraries. I was investigating this some time ago, but my poor desktop that was built on budget is not the best fit for training models, so iterations are way too slow for experimentation.

Sent you a DM!

And there is Finally, a Replacement for BERT: Introducing ModernBERT which claims better performance on code. So maybe it would be possible to embed both docs and each function source code and then use those for lookups!

It’s definitely possible, as there are already fine-tuned models that understand code pretty well.

I think it’s all about the way you decide to fine-tune and measure the efficiency, for me that honestly is still black magic as I’ve never delved into theory on this topic.

This also should be doable in elixir as bumblebee supports fine-tuning: Fine-tuning — Bumblebee v0.7.0

It’s interesting to consider whether RAG would actually beat full-text search (tool use) here. When I’m searching for docs my queries are almost always keyword searches, (library names, function names) - there’s not a whole lot of need for semantic lookups.

Again I really think the general solution will win here - you can do RAG on all of the docs in the world and still lose to a model that finds a blog post or forum thread which solves a given problem.

Interesting, I honestly think fine-tuning models is not the way to go, but rather create a system to provide the correct context to whatever LLM is the best at a given moment/in a given situation.

To that end, it should be ‘somewhat’ trivial to create a solution that always surgically inserts the newest docs of a given library when needed. Be it an MCP server that just fetches it, or a more sophisticated system that knows about the relevant libraries of a project, caches that info, and then retrieves the relevant sub-pages’ info when deemed relevant.

I think it depends on what you plan to be your end result.

I never cared about models guiding me on steps how to write code, but instead I want a flexible way to search for documentation. Either way, elixir core team said they are working on making such a tool happen, we will see what their end result will look like and the approach they decided to take.