HexDocs MCP - Semantic Search for Hex Documentation, Right in Your Editor ✨

This looks great, thanks for building and and contributing. MCP is a very (very!) new thing to me, and I’m behind the curve on AI-assisted development.

Do you (or anyone else) have any recommendations on how to write and structure library documentation to improve the quality of the embeddings that are derived from docs? I’m not sure I even phrased the question correctly but I’m optimistic you know what I mean!

I haven’t personally hopped on the AI editor trend, but a lot of people have been asking for stuff like this, so nice work

I think you’re looking for this. I wonder, if you had used your tool to find this documentation, could it have improved itself?

Hey, great question! I think I get what you’re asking—are you thinking specifically about HexDocs MCP or just more generally? There are a lot of small optimizations the MCP server could make that would help, but they’d require a decent amount of work.

At a high level, I see this in two parts: the chunking mechanism and the search/re-ranking process.

For chunking, my current approach is pretty naive. I’m using TextChunker, which works with Markdown. That said, it’d be nice to have a smarter, more dynamic way to split content—maybe even something tailored specifically to HexDocs? I’m not totally sure what the best solution is here, but I definitely think there’s room for improvement. I’m still pretty new to all of this, so I haven’t nailed down a solid direction yet.

As for re-ranking, I think one of the most impactful things we could do is give RAG systems better signals about which embeddings should carry more weight. For example, if libraries included an llms.txt file—similar to what LangGraph suggests here—then tooling could prioritize content referenced in that file, treating it as more central or authoritative.

The tricky part, though, is how to maintain the llms.txt file as a library author. How do we decide what goes in it? How do we keep it up to date over time? I don’t have a good answer for that yet.

Hey, I actually considered this! I just wasn’t sure how to use Bumblebee alongside the stdio MCP server, which is written in JavaScript. I think it might be possible to drop the Ollama dependency if I used a native JavaScript library for embeddings—I just wasn’t sure which one to use. Ollama was super easy to get going since it works well with both Elixir and JavaScript, so I went with that for now. If you or someone could provide some direction here, I’d be open to a better solution!

@bradley, this is sweet, thanks for building this!

A few misc comments:

• Bumblebee really only makes sense to use if you’re willing to port the server to Elixir. Note that @thmsmlr has a (STDIO transport) MCP server implementation as part of Livebook Tools: livebook_tools/lib/livebook_tools/mcp_server.ex at master · thmsmlr/livebook_tools · GitHub

• This is a bigger discussion, and I’m not going to do it justice in this quick comment, but something to consider: using RAG for search, but not for context. RAG picks out relevant chunks and thus identifies relevant modules/docs, but separate LLM call(s) are made (with the entire contents of that module’s documentation, and a summary of the user’s ask), returning new LLM-summarized text that can then be passed off to the coding agent. See ReAG: Reasoning-Augmented Generation for a better explanation (although that’s not exactly what I’m proposing, but it’s along the same lines).

My preferred tools don’t yet support MCP, so I haven’t played around too much, but the above is an approach I’ve been planning on playing around with.

Thanks for the excellent read! TIL ReAG.

One decision I made when designing this library was to limit the number of retrieval results to avoid overwhelming the LLM’s context window. However, after reflecting on your points, I’m considering an alternative approach: introducing an intermediate filtering step after the embedding retrieval but before delivering the final results.

Specifically, the idea would be to retrieve a broader set of results initially, then use the LLM itself to iterate through these and apply an isIrrelevant: true filter, discarding entries that aren’t contextually relevant. Although this would likely increase response latency, I believe it could significantly enhance the relevance and overall quality of the results.

What do you think of this approach?

One concern I still have relates to the chunking mechanism itself—since valuable context could be lost within individual chunks, it would be ideal to remove the need for chunking altogether which the blog post mentions.

Yes, that’s the general the idea. I would experiment with keeping the chunking
(but strictly for search), and then sending each document that the chunk was excerpted from for in its entirety and have the LLM provide back the relevant subset of context. (Which may need to happen across parallel calls, depending on the number and size of documents.)

The challenge with the isIrrelevant approach is not so much the flagging of what’s relevant or not, but rather that the initial vector similarity search is probably not going to pick the best “chunks”; i.e., you’ll get results that are similar to the user’s query, but not the best ones for actually solving their problem.

But yes, the cost is a lot more latency (and tokens). Might be worth exposing a “fast” tool that exclusively does RAG, and another tool that takes this alternative approach.

(One other thought: I don’t think MCP supports returning another tool call, or prompt(s) from a tool call, but if it did that could be a neat way to implement ReAG, as it gives more guidance/control back to the MCP client.)

I was thinking specifically about what documentation should be written to maximise the chances of a developer getting a relevant and useful result from MCP-based searches.

For example, most of my libs have reasonable API documentation. But they definitely lack in tutorial type documentation. If the trend is towards more AI-assisted help then perhaps I need to be paying more immediate attention to different types of documentation.

Off hand these are my thoughts but I’m curious if anyone else agrees/disagrees.

Order of importance:

1. Force runtime errors/warnings at compile time wherever possible
2. Minimize ambiguity via types, be it via behaviours, specs, etc.
3. If none of this is possible, provide an examples via documentation/doctests
4. Add an llms.txt as mentioned above so devs could use mcpdoc or similar

Hey all! I’ve put out v0.2.0-rc.1 but am waiting to release it until I’m more confident in how well it works. Is anyone willing to try it out/give me feedback? I would be eternally grateful!

The main updates are:

• Added a fetch command to allow the mcp server to auto fetch docs
• You no longer need to add hexdocs_mcp to your mix.exs. Reasons for this were:
  1. To reduce toil and having to keep dependencies up-to-date for a non-critical tool
  2. I like to switch between various projects and having to remember/install in every project isn’t ideal for my workflow to run mix hex.docs.mcp fetch ...
  3. To make installation as easy as possible for folks. Theoretically you don’t even need to have elixir installed to use the mcp server, though I haven’t tested this.

You can see all of the details and a migration guide in the release notes.

I will admit, it feels odd that the primary entrypoint to this project is a node app, but I think it’s right for now. Once the MCP ecosystem evolves and is more stable, migrating to an elixir MCP server would make a lot of sense I think. I’m open to ideas here if you have any!

I went ahead and pushed v0.2.0. It’s the same as the rc version I mention above. Check out the changelog/readme for details!