RAG app using Elixir. Feasible?

Apologies for hijacking this thread, but here’s another blog post that shows how you can implement a local RAG system in Elixir.

I want to share this here since it’s RAG related:
We implemented the Jina embeddings v2 model in Elixir so we can use it for our RAG project.
I hope this blog post also helps you in case you want to use a model that’s not supported out of the box in Bumblebee.

Just wanted to give a little update:

We ended up abstracting the text chunking part of our RAG app out into an open source package and it has been published on hex.

https://github.com/revelrylabs/text_chunker_ex

While the text chunking is definitely not the hardest part of all of this, it was the easiest thing for us to abstract out to share with the community. Feel free to check out the announcement blog post.

This is definitely more than an experiment. We are building an application called ProdOps.AI which has the goal of augmenting software/product delivery teams using generative AI. Its very much in its early stages, primarily being used by our internal teams and close partners, but we plan to release it out in the wild in the very near future. Don’t want to promote that here, but just for context the primary use cases (currently) include generating product roadmaps, backlogs, user stories, and implementation plans augmented with proprietary data that can be either synced up via an external data source (e.g. github/slack/google drive) or manually uploaded. We also built a pretty useful prompt template management system which allows for building prompts that can query against a given organization’s proprietary data based on user inputs at the time of generation. We initially created this prompt management system to enable our team to easily iterate on prompts for specific use cases, but it turned into a pretty useful generic RAG-based prompt management system. We hope to abstract some of the less opinionated/product development specific stuff out into open source tooling at some point, but that will take some time and effort.

As far as fine tuning goes, we havent crossed that bridge yet. So far, we have gotten what we needed from RAG + GPT4 without fine tuning. We definitely will be digging more into fine tuning soon, but it doesnt seem necessary for this application at this juncture.

That said, I asked my team if they had any opinions on training LoRAs etc, and was reminded that Sean Moriarity mentioned in a talk a while back that he’d fine-tuned a model using Python tooling but then deployed it on Elixir for production inference. Check his talk out (should be timestamped where he talks about the fine tuning)

Very interesting experience indeed. Many thanks for sharing it.
Did you use this in production or just an internal experiment ?
How did you perform the finetuning (PEFT/LoRA) of your model(s) ? Have you been also using Elixir for that?

For what its worth, my team at Revelry has been building a custom RAG-based application with Elixir/Phoenix over the past year, and I am really happy we decided to stick with Elixir.

I gave a brief breakdown of the steps needed to build a RAG flow (without LangChain) in this blog post (see the section about “How to build a RAG flow”). That article is primarily about comparing OpenAI’s API offerings, but the tangent about RAG that I gave is relevant, especially given that we did exactly that using Phoenix and Elixir. We are currently using OpenAI for our LLM, but the RAG part is really LLM agnostic. We could plug in open source models in place of GPT 4 if we wanted to, but right now it is getting us the best results.

Directly from the blog post linked above are the general steps to build a RAG flow:

• Set up a Vector Database
  • Options range from building it yourself in Postgres with PG Vector, to open source VectorDB’s such as Chroma, to a nice managed solution like Pinecone.
• Enable uploading of documents to your system that need to go into the vectorDB (probably via some web interface)
• Extract plain text from the files (can be more involved depending on the file type)
• For each uploaded document, chunk the text based on content type
  • there are a lot of decisions to be made here in terms of how large the chunks are, what to separate the chunks on, how much overlap there should be, etc
• Convert those chunks into vector embeddings
  • You can use openAI’s embedding models via API, but you can also use any embedding model of your choice (open source or proprietary).
• Store those vector embeddings in your vector DB
• Query against the vector DB using semantic search to pull relevant pieces of information out, and inject that info into a prompt before it’s sent to the LLM
• Send the “retrieval augmented” prompt to the LLM to generate the stuff. Hence “Retrieval Augmented Generation”.

Thanks Jose, obviously it is feasible RAG with Elixir.
I’m facing the big challenge in persuading the customer to adopt this new (and exotic) approach, as they prefer to stick to tools and technologies that are “well-established”, “industry-tested” and take no risks.

This is a great example. I will try to contact fly.io maybe get some insights on their experience.

Oh, definitely. If you have PDFs, you need to extract text from them for embeddings. Once you choose a LLM, you need to consider prompt engineering, window size, etc. Some models make it easier than others.

But those considerations all exist around the three main blocks I have mentioned and will vary per use case and per technology. That’s another reason why I would start with something off the shelf and then break it apart based on your needs.

There are a few more steps usually involved that wrap around LLM, either on the indexing, retrieval or formulation of the responses phase and I am not sure how much of these tools we have in our ecosystem. If you look at LlamaIndex, for example, you can pick from several strategies for querying/retrieval, pre-processing and post-processing of data, summarization steps, verification of alignment steps, context/window tracking, logging and such. They work, out of the box, for the most part.

I think it’s feasible to build an RAG tool in plain Elixir but you have to be prepared to build more of these building blocks yourself.