The future for generating sql will be GitHub - elixir-dbvisor/sql: SQL provides state-of-the-art, high-performance SQL integration for Elixir, built to handle extreme concurrency with unmatched expressiveness and ergonomic query composition. Write safe, composable, parameterized queries directly, without translating to Ecto or any ORM. · GitHub
By using the basic engineering principle of separation of concern you can built something very powerful with Enumerable and Collectable protocols.
Really cool idea using Enumerable and Collectable directly on queries and implementing the protocol right on the Ecto.Repo.
So, it’s not that a persistent connection is required so much as that it’s preferable. In order to query a table you have to open it first. That incurs some overhead, but I can store the connection reference in a rustler ResourceArc to share the open table back to the BEAM.
This isn’t documented, but is something that a LanceDB team member told me on discord in response to a very direct question about it.
Hot diggity. Yeah - users do not connect to LanceDB over tcp at all. You can have basically limitless reads and appends, and you only want to throttle concurrent update/delete (mutation) operations to a recommended 10 max. The exqlite discussion you linked is really useful because it seems like similar dynamics are at play.
I am not sure where I got the impression that I needed to implement DBConnection to create an Ecto Adapter but I’m going to regroup and try to attack this directly as an Adapter.
It’s not a noSQL DB, fwiw. “LanceDB supports a growing list of SQL expressions.” including DataFusion math functions. However it’s not relational, i.e. there are no joins.
It also has operations like “nearest_to” for vector similarity search, as well as full-text search, and hybrid “vector + full-text” search with configurable reranking strategies. Very good for RAG. So if I can make an Ecto adapter that supports that kind of stuff either via options or some DSL, that will be ideal.
Thank you. I’m marking this as the answer because it answered the question I didn’t even think to ask: do I need DBConnection for this use case? The answer seems to be no, mostly because LanceDB doesn’t need to connect over TCP. I’m going to look into creating an Ecto Adapter instead.
Not really looking to pair program at the moment. Really just trying to wrap my head around what I need to do to plug into the Ecto ecosystem and it’s looking like I won’t actually need to implement DBConnection to make an Ecto adapter. Woohoo! I definitely agree LLMs can be great learning tools etc.
Yea I don’t think DBConnection is necessary for you. Even for the whole “opening a table” concept you could have some sort genserver / ets table in the background that handled this. Just make sure to scope it to a particular repo.
I am potentially interested in chatting in the EEF slack! Not sure how to join it though.
Hi @eileennoonan
As a vector DB, it implements Arrow Flight, which should be more efficient for reading.
Have you tried connecting with ADBC (GitHub - livebook-dev/adbc: Apache Arrow ADBC bindings for Elixir · GitHub) ?
