Oh, definitely. If you have PDFs, you need to extract text from them for embeddings. Once you choose a LLM, you need to consider prompt engineering, window size, etc. Some models make it easier than others.
But those considerations all exist around the three main blocks I have mentioned and will vary per use case and per technology. That’s another reason why I would start with something off the shelf and then break it apart based on your needs.
Thanks Jose, obviously it is feasible RAG with Elixir.
I’m facing the big challenge in persuading the customer to adopt this new (and exotic) approach, as they prefer to stick to tools and technologies that are “well-established”, “industry-tested” and take no risks. 
For what its worth, my team at Revelry has been building a custom RAG-based application with Elixir/Phoenix over the past year, and I am really happy we decided to stick with Elixir.
I gave a brief breakdown of the steps needed to build a RAG flow (without LangChain) in this blog post (see the section about “How to build a RAG flow”). That article is primarily about comparing OpenAI’s API offerings, but the tangent about RAG that I gave is relevant, especially given that we did exactly that using Phoenix and Elixir. We are currently using OpenAI for our LLM, but the RAG part is really LLM agnostic. We could plug in open source models in place of GPT 4 if we wanted to, but right now it is getting us the best results.
Directly from the blog post linked above are the general steps to build a RAG flow:
Very interesting experience indeed. Many thanks for sharing it.
Did you use this in production or just an internal experiment ?
How did you perform the finetuning (PEFT/LoRA) of your model(s) ? Have you been also using Elixir for that?
This is definitely more than an experiment. We are building an application called ProdOps.AI which has the goal of augmenting software/product delivery teams using generative AI. Its very much in its early stages, primarily being used by our internal teams and close partners, but we plan to release it out in the wild in the very near future. Don’t want to promote that here, but just for context the primary use cases (currently) include generating product roadmaps, backlogs, user stories, and implementation plans augmented with proprietary data that can be either synced up via an external data source (e.g. github/slack/google drive) or manually uploaded. We also built a pretty useful prompt template management system which allows for building prompts that can query against a given organization’s proprietary data based on user inputs at the time of generation. We initially created this prompt management system to enable our team to easily iterate on prompts for specific use cases, but it turned into a pretty useful generic RAG-based prompt management system. We hope to abstract some of the less opinionated/product development specific stuff out into open source tooling at some point, but that will take some time and effort.
As far as fine tuning goes, we havent crossed that bridge yet. So far, we have gotten what we needed from RAG + GPT4 without fine tuning. We definitely will be digging more into fine tuning soon, but it doesnt seem necessary for this application at this juncture.
That said, I asked my team if they had any opinions on training LoRAs etc, and was reminded that Sean Moriarity mentioned in a talk a while back that he’d fine-tuned a model using Python tooling but then deployed it on Elixir for production inference. Check his talk out (should be timestamped where he talks about the fine tuning)
Just wanted to give a little update:
We ended up abstracting the text chunking part of our RAG app out into an open source package and it has been published on hex.
https://github.com/revelrylabs/text_chunker_ex
While the text chunking is definitely not the hardest part of all of this, it was the easiest thing for us to abstract out to share with the community. Feel free to check out the announcement blog post.
I want to share this here since it’s RAG related:
We implemented the Jina embeddings v2 model in Elixir so we can use it for our RAG project.
I hope this blog post also helps you in case you want to use a model that’s not supported out of the box in Bumblebee.
Apologies for hijacking this thread, but here’s another blog post that shows how you can implement a local RAG system in Elixir.