MosaicDB - a hobby project that uses Nx (distributed SQL vectordb rag)

Elixir Framework Forums Nx Forum
, , ,
1

I wanted to stop by to say thank you

I was recently working through a hobby learning project for retrieval augmented generation, with some bells and whistles ( elixir is not my first language )

It’s probably reinventing the wheel, but hey, wheels can be interesting:

And well, Nx made it so easy to do!

So, again, Thanks!

12 Likes
2

Really cool project! Kudos!

3 Likes
3

Looks very cool!

I’ve had an idea for a while now that you could get passable vector search with Nx using xor and popcnt on 1-bit quantized vectors. For those unaware, xor/popcnt are essentially 1-bit cosine distance.

Instead of building an index the idea is that you just brute-force search very quickly with native vectorized instructions, which I figure Nx should be able to do in batches without eating any significant Elixir speed penalty. Unlike vector indexes (e.g. HNSW) this approach has the advantage of being extremely simple and there is no need for external dependencies (just Nx) which I find appealing.

I don’t know all that much about vector search and I haven’t kept up with the state of things for a few months (and I know things are changing that fast). Do you have any thoughts on this?

1 Like
4

I like the idea, I haven’t personally tried it out, I’ve tried out some of the other configurable distance types via redis vector search, but didn’t see much of a difference for top k for the data types, I’ve been working with.

Were you thinking just FLAT index, with xorpopcnt?

I believe there is precedence? Somewhere in the features in qdrant maybe?: qdrant/lib/quantization/src/encoded_vectors_binary.rs at e293ab1f08d3eb8fb8d9c3380c1de71b7a76570c · qdrant/qdrant · GitHub

Something, like that?

I need to do some more reading on it.

1 Like
5

I see Redis has a “FLAT” index, and I think it’s what I’m talking about but it’s hard to tell because their docs don’t explain what it does at all (unfortunate). What’s weird about that name is a) the brute-force approach is kinda not an index by definition and b) there is a type of vector index called IVFFLAT but I guess this is just a case of unfortunate naming?

But yeah, my understanding is that up to a certain dataset size xor/popcnt on 1-bit quantized vectors as a first pass is actually surprisingly performant. Then you can do reranking either with cosine similarity or, apparently, LLM reranking is pretty good. The benefit of this approach is simplicity; HNSW and co are extremely complicated and expensive to update incrementally. Keeping everything in RAM helps but then you have to keep everything in RAM.

Keep in mind my knowledge is limited and probably like 6mo out of date here so I have no idea what the SOTA is with on-disk vector indexes right now. I think there has been progress.

But as far as “shortest path to a passable pure(ish)-Elixir vector search” this seemed to be the way to go. Of course I have yet to pursue this.

1 Like