RAG app using Elixir. Feasible?

This talk might be useful as it examines the eco system an features that make Elixir powerful for MLops.

And this:

Tx, I was aware about these. They are very promising presentations but are just initiatives.
(I can’t propose a project based on just conference presentations. The client’s CTO will need to see “facts”).
Wondering if some of you have actual experience on it at the business/production level.

I’m also interested in RAGs & systems to query/chat with a document collection. Have been testing PrivateGPT - hope to find an Elixir equivalent.

I am aware of PrivateGPT, I’ve been playing a bit with it. But It’s only RAG (chat with documents)
Our client’s needs include prior fine-tuning/PEFT (LoRA etc.) of the model for his specific domain. A complete tool for this is LangChain.
Regrettably, our trials reveal that LangChain, limited to Python/JS deployments, falls short in large-scale deployment for multiple concurrent clients/agents.

Have you tried working with GitHub - brainlid/langchain: Elixir implementation of a LangChain style framework that lets Elixir projects integrate with and leverage LLMs. · GitHub ??

I know it. Not usable for me as it deals only with GPT API and their proprietary models ($$$).
Our client prefers an implementation based solely on open-source models.

Besides.. no offence, but this Elixir implementation of LangChain is very limited (still too young for a production deployment).

It’s a pity because Elixir (running on Erlang’s BEAM)) is the ideal platform (IMHO) for such AI applications. I don’t see much interest around to extend it in this direction.

I would suggest actually taking the models you are wanting to use for a spin using Livebook.

Right now today you can import models developed in say python and operationalize them using Elxir with Bumblebee, Ortex, axon_onnx, Axon, Scholar etc all underpinned by NX.serving which can provide distributed serving of models using every GPU in your cluster.

You can’t make robust decisions from the armchair and will need to do some validation for yourself. Whilst Elixir is realitively new to the ML space it has the underpinnings to be the compelling solution for deployment. The weakest area currently is model development, but for serving models Elixir has tools to import existing models. That’s not to say there are not gaps but the important thing to identify is are there untenable gaps for the use cases you currently have and if you do use Elixir then what parts of your current enviornment will it replace. I would hazard to guess you would pick a scoped part of the overall solution and try Elixir there, get some experience then consolidate and expand from that.

One of the most advanced intellectual property search systems migrated all their models and processing to Elixir to operationalize their solution. In doing so they halved their AWS costs through less complexity processing 100’s of millions of patents on a weekly basis when updating their models. Their original talk is here:

Their latest talk is here:

It is definitely possible. A RAG system has three components:

1. Models for generating embeddings
2. An index
3. A LLM

You will find support for generating embeddings in Bumblebee. You need to pick a model though and sbert is a starting point: https://www.sbert.net/

Indexes is the area we have least developed on. There are both ExFAISS and hnswlib bindings on GitHub. We want to officially release the latter at some point. Alternatively, you can pick a vector database or even PG with pg_vector for this step, which I would recommend.

Then you need to pick a LLM, either with Bumblebee or off the shelf.

Here is a post, a bit dated, that gives you more pointers: Semantic Search with Phoenix, Axon, Bumblebee, and ExFaiss - DockYard

Honestly, implementing this has both technical moving parts but business building parts. What is the best model for your use case? Best embeddings? How to generate embeddings for your documents? Etc. my suggestion would be to pick an off the shelf solution to evaluate the results and build a prototype, and only then evaluate what makes sense to bring in-house for performance, value, security reasons.

In case it matters, I am speaking both as a library author and as someone who has built more than one proof of concept RAG system.

Here is an article that shows how to implement step 1 and 2 with Elixir: Real World ™ Machine Learning on Fly GPU's · The Phoenix Files

There are a few more steps usually involved that wrap around LLM, either on the indexing, retrieval or formulation of the responses phase and I am not sure how much of these tools we have in our ecosystem. If you look at LlamaIndex, for example, you can pick from several strategies for querying/retrieval, pre-processing and post-processing of data, summarization steps, verification of alignment steps, context/window tracking, logging and such. They work, out of the box, for the most part.

I think it’s feasible to build an RAG tool in plain Elixir but you have to be prepared to build more of these building blocks yourself.