RAG app using Elixir. Feasible?

For what its worth, my team at Revelry has been building a custom RAG-based application with Elixir/Phoenix over the past year, and I am really happy we decided to stick with Elixir.

I gave a brief breakdown of the steps needed to build a RAG flow (without LangChain) in this blog post (see the section about “How to build a RAG flow”). That article is primarily about comparing OpenAI’s API offerings, but the tangent about RAG that I gave is relevant, especially given that we did exactly that using Phoenix and Elixir. We are currently using OpenAI for our LLM, but the RAG part is really LLM agnostic. We could plug in open source models in place of GPT 4 if we wanted to, but right now it is getting us the best results.

Directly from the blog post linked above are the general steps to build a RAG flow:

• Set up a Vector Database
  • Options range from building it yourself in Postgres with PG Vector, to open source VectorDB’s such as Chroma, to a nice managed solution like Pinecone.
• Enable uploading of documents to your system that need to go into the vectorDB (probably via some web interface)
• Extract plain text from the files (can be more involved depending on the file type)
• For each uploaded document, chunk the text based on content type
  • there are a lot of decisions to be made here in terms of how large the chunks are, what to separate the chunks on, how much overlap there should be, etc
• Convert those chunks into vector embeddings
  • You can use openAI’s embedding models via API, but you can also use any embedding model of your choice (open source or proprietary).
• Store those vector embeddings in your vector DB
• Query against the vector DB using semantic search to pull relevant pieces of information out, and inject that info into a prompt before it’s sent to the LLM
• Send the “retrieval augmented” prompt to the LLM to generate the stuff. Hence “Retrieval Augmented Generation”.

It is definitely possible. A RAG system has three components:

1. Models for generating embeddings
2. An index
3. A LLM

You will find support for generating embeddings in Bumblebee. You need to pick a model though and sbert is a starting point: https://www.sbert.net/

Indexes is the area we have least developed on. There are both ExFAISS and hnswlib bindings on GitHub. We want to officially release the latter at some point. Alternatively, you can pick a vector database or even PG with pg_vector for this step, which I would recommend.

Then you need to pick a LLM, either with Bumblebee or off the shelf.

Here is a post, a bit dated, that gives you more pointers: Semantic Search with Phoenix, Axon, Bumblebee, and ExFaiss - DockYard

Honestly, implementing this has both technical moving parts but business building parts. What is the best model for your use case? Best embeddings? How to generate embeddings for your documents? Etc. my suggestion would be to pick an off the shelf solution to evaluate the results and build a prototype, and only then evaluate what makes sense to bring in-house for performance, value, security reasons.

In case it matters, I am speaking both as a library author and as someone who has built more than one proof of concept RAG system.

Just wanted to give a little update:

We ended up abstracting the text chunking part of our RAG app out into an open source package and it has been published on hex.

https://github.com/revelrylabs/text_chunker_ex

While the text chunking is definitely not the hardest part of all of this, it was the easiest thing for us to abstract out to share with the community. Feel free to check out the announcement blog post.