eileennoonan
Seeking guidance implementing DbConnection
I’m working on an Elixir client for LanceDB (sqlite for vector storage) called ElixirLanceDB. Most of this work so far has comprised of writing Rustler NIFs over the LanceDB rust client. I’ve got enough of the library implemented that I can use it for a basic RAG / hybrid search server, and I plan to keep fleshing it out steadily and would eventually like to release it on Hex.
I’d like to implement DBConnection so that I can plug it into Ecto and AshDataLayer and the rest of the ecosystem.
The problem I’m running into is, I don’t understand a lot of these DBConnection callbacks, and they don’t all seem to apply to LanceDB in the same way as they’d apply to Postgres/MySQL/SQLite etc.
For example, a connection to the database itself is really only used for opening connections to tables. All the querying etc happens on the table. So I’d actually want to maintain connection pools both to the DB itself as well as to individual tables.
I’d want to be able to plug into migration systems, but I probably also want the ability to create tables on the fly.
I have a pretty good handle on the LanceDB rust client at this point, but I’m a little overwhelmed by this DBConnection aspect. I guess my big broad question here is, can anyone give me some pointers or suggest some reading or otherwise help me wrap my head around this? Thanks!
Marked As Solved
Schultzer
You might not even need DBConnection. Since LanceDB can speak SQL then you could maybe try shim it with Ecto.SQL.Adapters.SQL.
Or look at GitHub - Schultzer/ecto_qlc: QLC-based adapters and database migrations for Ecto · GitHub or GitHub - foundationdb-beam/ecto_foundationdb: FoundationDB Adapter for Elixir's Ecto · GitHub for none DBConnection Ecto adapters.
Also Liked
jstimps
There’s a lot of things I learned (and still learning!) about implementing an Ecto Adapter that aren’t written down anywhere. I’ve been meaning to write a blog post about the experience, but I don’t even have a blog yet, so that’s still on the project list.
One of those things was realizing I shouldn’t use DBConnection for my particular backend. I don’t know enough about LanceDB to make a call, but if it were me, I’d start without it and add it back in later if needed.
A bit off topic, but this is actually something I’m wrestling with right now on my Ecto Adapter, so I’m curious about what’s in your future for your project.
Aside: I don’t know if folks would be interested in connecting on the EEF Slack for more realtime collab but I’d be up for it.
LostKobrakai
I’m wondering if you’re to deep into the weeds here. Yes there is low level details of connection handling. That’s not the only thing to deal with though when ecto is meant to interact with a db. I don’t know much about lancedb, but my point about looking at the higher level was mostly about the question if sql is actually sufficient or if there needs to be other querying apis and stuff like that. If it’s sql, then what sql statements are required to be supported, does Ecto.Query support building up all the necessary statements, do migrations support what is needed, …
DBConnection is also not at all required to build an ecto adapter. It just happens to encapsulate a lot of pooling requirements that you encounter when connecting to a database “server” over tcp. It might or might not be the right tool for a local database, which comes with different tradeoffs and limitations.
eileennoonan
Hot diggity. Yeah - users do not connect to LanceDB over tcp at all. You can have basically limitless reads and appends, and you only want to throttle concurrent update/delete (mutation) operations to a recommended 10 max. The exqlite discussion you linked is really useful because it seems like similar dynamics are at play.
I am not sure where I got the impression that I needed to implement DBConnection to create an Ecto Adapter but I’m going to regroup and try to attack this directly as an Adapter.
It’s not a noSQL DB, fwiw. “LanceDB supports a growing list of SQL expressions.” including DataFusion math functions. However it’s not relational, i.e. there are no joins.
It also has operations like “nearest_to” for vector similarity search, as well as full-text search, and hybrid “vector + full-text” search with configurable reranking strategies. Very good for RAG. So if I can make an Ecto adapter that supports that kind of stuff either via options or some DSL, that will be ideal.
Last Post!
eileennoonan
Yup, I’m familiar with that thread and it’s a big part of why I’m not planning to use DBConnection anymore
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"