allen-munsch
MosaicDB - a hobby project that uses Nx (distributed SQL vectordb rag)
I wanted to stop by to say thank you
I was recently working through a hobby learning project for retrieval augmented generation, with some bells and whistles ( elixir is not my first language )
It’s probably reinventing the wheel, but hey, wheels can be interesting:
https://github.com/allen-munsch/mosaic
And well, Nx made it so easy to do!
So, again, Thanks!
Most Liked
seanmor5
jstimps
Not sure if this will be helpful, but I can share my experience so far.
I’m still on this journey, more of a side quest. I have a working implementation of HNSW in Erlang, using FDB for storage, and Nx for vector operations. Benchmarked against the Irisa Sift datasets, recall is good, but predictably performance is quite poor. Having to do hundreds of network round trips for graph traversal is a death knell. I knew this going in, but I wanted to see it for my own eyes. If only there was an embedded data store that I could drop in as a replacement… 
I’m not up to date on the SOTA, but I am interested in continuing my side quest by studying SPANN+SPFRESH. I’m sympathetic to the idea that the vector store is changing dynamically over time rather than a static graph computed by some periodic job, as is typical elsewhere.
I also believe that the need for optimizing compute over a graph will only increase, both in hardware accelerated compute and also in database query execution, and perhaps even some combination of the two someday.
jstimps
Here’s the datasets I was referring to
http://corpus-texmex.irisa.fr/
My comment about embedded key value stores was made as a reference to some recent new projects announced on the forum, including Hobbes and Bedrock. Both are projects I’m following closely.
Regarding your new approach, for my own use cases for ANN, I wouldn’t be able to accept the heavy compromise on recall. But perhaps there are other use cases where that’s ok.
Last Post!
garrison
Consensus algorithms (vsr/paxos/raft/etc) are the only tool for fault-tolerant strong consistency.
The bug in your example is that in consensus the leader is not allowed to reply before replicating to a quorum, all of whom must persist to disk. Of course there is nothing database companies love to do more than lie about guarantees, and the exceptions (e.g. Tigerbeetle) are rare.
But this is not a technical problem.
Popular in Discussions
Other popular topics
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"