I’m working on using Rustler to create Elixir bindings for this extremely nifty embedded multimodal AI / vector database LanceDB. Lance is massive, but I already know I could go very far with maybe a couple dozen NIFs.
I’m still pretty new to Rust/Rustler, so I have a couple very broad questions for folks.
First, what is your workflow like with Rust/Rustler? How does one go about getting rapid feedback from Rust? With Elixir I can very quickly open iex or ExUnit. In Rust I find myself in this double-bind where the function won’t compile, so I want to inspect the data, but I can’t because the function won’t compile…
Then Rustler adds another layer on top of this, where maybe the code would work in Rust, but I’m not returning the correct type for Rustler so it still won’t compile.
Second, this morning I’m just trying to wrap my head around what it would take to return a LanceDB Connection to Elixir. I believe I’ll want to use a ResourceArc for this. Am I right in thinking that I’ll need to implement an encoder/decoder for the lancedb::Connection struct?
Creating a lancedb::Connection is also an asynchronous operation - are there any resources on dealing with async Rustler?
I’m quite excited about this project because LanceDB is incredible and I think the Elixir world deserves easy access to it! I’m just very disoriented and I feel like a little advice and maybe even coaching would speed this up a ton.
Not much of help, but I’ve wanted to explore two separate topics with rustler recently and also hit the lack of explanations of what needs to be done to create resources. I’d love to know more how this is intended to work as well.
I will have something for both of you pretty soon. After more than 4 years, I have finally started reviving my SQLite Elixir<=>Rust bridge library and with the help of several people and an LLM I found the “blessed” ways to do several things – including the non-obvious gotcha when you think you are returning a binary but it turns out to be a list of integers on the Erlang / Elixir side.
Is it truly Rust-async btw? As in, you need the tokio runtime compiled in? Or is it something that returns a handle and just delegates work to a background thread? If the former, I haven’t yet made strides there but I remember somebody on the forum saying they learned how to pass Futures back to Elixir and resume / poll them later in the Rust code. I am sure we can dig it out in GitHub.
Polars (Rust) is the dataframe layer for Elixir Explorer , incorporated via rustler. So there might be useful code in Explorer to teach about using Rustler (I checked with ChatGPT that I wasn’t telling you rubbish … if that’s encouraging).
To be a little less cryptic: I have deferred the pool management to the Rust side and used DashMap for its minimal locking features (the map is a key-value store of pools). My pool-of-pools works roughly like this: using an ID (an u64 with values dispensed by an AtomicU64, again to avoid global locks) as keys and the actual pool / pooled connection as the value in the map. Then you have just a few functions that use a static DashMap variable (a global var), wrapped in OnceLock – to achieve builtin (stdlib and not requiring external library) initialize-once semantics.
Then your own “connection handle” can be something like this:
i.e. a simple tuple struct because otherwise the compiler will yell at you that the “foreign” type does not implement the Encoder trait. So you wrap it in your own type that you control and specifically encode / decode the pieces of data you care about.
It’s a bit paradoxical and I can’t say I get it fully f.ex. in the example above u64 should be freely serializable but I couldn’t just return NifResult<u64>. But when I did the above code snippet, it worked.
Anyhow, I am working on a PR for my project and will have code to link to VerySoon™ now.
This is all incredible stuff to get started with, thanks very much everyone!
Is it truly Rust-async btw? As in, you need the tokio runtime compiled in? Or is it something that returns a handle and just delegates work to a background thread?
Not sure - I’m just not that knowledgeable on it yet. I can say that Lance is using Tokio internally when opening DB connections, and that I need to .await the result of that function, and so I need to mark my function as async.
Definitely looking forward to seeing the blessed path! SQLite would be an awesome reference point since it’s also an embedded file DB.
Polars (Rust) is the dataframe layer for Elixir Explorer , incorporated via rustler. So there might be useful code in Explorer to teach about using Rustler (I checked with ChatGPT that I wasn’t telling you rubbish … if that’s encouraging).
This will definitely be helpful! I’m also referencing the official Lance NodeJS implementation quite a bit - it uses NAPI-rs.
My PR is in and I am extremely ashamed of it. At the end I just got pissed and wanted stuff to compile so chose a very clunky approach to return ok/error tuples. It will be reworked very soon, so please don’t copy that!
But you can take inspiration in how to have a global pool of pools with minimum locking (only when a new connection is opened i.e. when an entry in the map is created; EDIT: sorry, also when the connection is closed i.e. when an entry in the map is removed). And how to get around the various encoding / decoding troubles.
But again, please don’t take my return values implementation seriously. It’s crap.
Here’s the link to the file. It’s 252 lines as of the time of this comment. I encourage you to read the PR’s description as well.
@filmor If you can spare a little time I’d appreciate criticisms, no matter how scathing they are. My Rust became rusty and I am gradually catching up lately.
Encoders/decoders and resources are very different beasts.
A type that implements Encoder and/or Decoder will get translated when passing from the BEAM to Rust (Decoder) or from Rust to the BEAM (Encoder). Low-level this means that a new object is generated whenever passing over from one to the other (with a small set of exceptions, but that can be regarded as an implementation detail).
A resource type “stays” in Rust and only an opaque reference is passed from Rust to the BEAM.
You want resources when using types that “live” in Rust (e.g. if they manage files, network connections, etc.).
I refactored the way how resources are implemented recently due to Rust language changes. The “new” way is to implement the Resource trait combined with the resource_impl attribute macro:
#[rustler::resource_impl]
impl rustler::Resource for Connection {}
With this, explicit registration is not necessary anymore and you can just start using ResourceArc<Connection> in your NIFs. Just beware that the type has to be Sync.
I know but I needed env for something that actually got removed later and kind of stuck with the old approach since. My main interest is how to have my own *Result type that gets transparently returned from Rust to Elixir without doing all the .encode(env) dances. Note that just having Encoder for it didn’t work…
But as said, I got pissed and just wanted to “ship”. Now I really want to make the thing as per modern Rustler and your recommendations.
Is the #[rustler::resource_impl] thingy documented somewhere else outside the official docs? I admit they confused me a bit.
rusqlite::Connection is Send but not Sync, hence the entire dance with DashMap and returning keys that point to stuff inside of it in my code (and never returning the actual connection wrapped in ResourceArc).
At the moment I’m really liking @dimitarvp approach of just managing a pool of connections in a Rust hashmap, and only passing some atomic ID back to the BEAM.
With this, explicit registration is not necessary anymore and you can just start using ResourceArc<Connection> in your NIFs. Just beware that the type has to be Sync.
That is an awesome refactor by the look of it.
Ok so, establishing a DB Connection is itself an async operation in rust. When you say the type has to be Sync, can I accomplish that using something like tokio and block_on?
Sorry to be too pedantic: have in mind that it’s a 3rd party library (DashMap) that incorporates a RwLock so mutating the map does an exclusive locking while not locking anything when reading. It’s not Rush’s builtin HashMap.
I will also try doing the same with evmap because it’s advertised to be faster than DashMap in read-heavy workflows (which is exactly the case for database wrappers IMO; we only occasionally open/create connections but access them much more often. (this is copied from my PR’s description).
OK, so compiling in the async runtime it is. IMO start a PR and add a few (currently failing) tests so me and others can take a look and contribute directly. This is of interest for me so I might try my hand at it – but I don’t want to supply the entire test harness before that.
If I have to do async code on the Rust side, I would just write a full Async Rust application and communicate with it through a port. There will be some overhead though. Synchronous Rustler is a well trodden path, so is async communication via ports. I am afraid async Rustler is not.
Thanks for the mention ;). kuzu_nif needs some help w.r.t returning handles to connections. If you ever figure that out @eileennoonan with LanceDB, I’d love to learn too.
Well you can check my code in the PR linked above really. It’s really short, uses DashMap and an AtomicU64 and wraps the u64 in a Rust tuple struct so you can implement rustler::Encoder on it. That’s pretty much it.
There is a way and now I regret I never bookmarked the link to one fairly bright guy’s code that he posted here in ElixirForum. I’ll try to dig it in the next days. He was basically returning Future wrapped somehow to the Elixir code and you can then pass it back from Elixir to Rust where the Future resumes execution – or just gets polled, can’t remember which.
this is copied from my PR’s description).
