Rustler return and pass reference from Rust to Elixir and back to Rust

Sure, sorry. I dropped details over in the thread in the rust forums.

The problem was the lifetimes. I needed to replace any datatype that requires a lifetime, with the equivalent datatype that doesn’t (e.g. &strString, and &[&str]Vec<String>). I didn’t understand that lifetimes aren’t needed for what I’m doing. They’re only used at compile-time, so in my situation it doesn’t make sense to accept a lifetime parameter. For example, &str requires a known length at compile-time, but my string values are dynamic length.

So instead of…

struct SlidingWindow<'a> {
    map: HashMap<&'a str, Vec<Option<f32>>>,
    labels: &'a [&'a str],
    index: usize,
    length: usize,
    width: usize,
}

…we need to use…

struct SlidingWindow {
    map: HashMap<String, Vec<Option<f32>>>,
    labels: Vec<String>,
    index: usize,
    length: usize,
    width: usize,
}

And one other helpful thing. I realized you can package probably any mutable data in this generic structure:

struct Container {
    mutex: Mutex<MyStruct>,
}

rustler::init!(
    "Elixir.MyNif",
    [func1, func2],
    load = |env: Env, _| {
        rustler::resource!(Container, env);
        true
    }
);

So you put whatever you want into MyStruct, and you wrap it up for elixir like this…

let my_new_struct = MyStruct { ... };
let mutex = Mutex::new(my_new_struct);
let container = Container { mutex };
Ok(ResourceArc::new(container))

…then you can receive the same thing back, lock the mutex, unpack it, and do whatever mutations you need.

#[rustler::nif]
fn print(arc: ResourceArc<Container>) {
    arc.mutex.lock().unwrap().print();
}

I’m happy to say that I have everything working in my project, have a look at master if you’re interested.

3 Likes

Oh, I did exactly the same for my sqlite3 library! Glad that you found the idiom!

Sigh, I should start blogging on these topics. I know it will help a lot of people.

3 Likes

Thanx @rm-rf-etc, @dimitarvp
This code work for me. I’m sqlite memory for data store.
this my code.

pub struct MyStruct {
    pub a: i64,
    pub conn: Connection
}
//
fn load(env: Env, _info: Term) -> bool {
    rustler::resource!(Container, env);
    true
}

struct Container {
    mutex: Mutex<MyStruct>,
}

#[rustler::nif]
fn add(a: i64, b: i64) -> ResourceArc<Container> {
    let conn = sqlite::open(":memory:").unwrap();
    let my_new_struct = MyStruct {a: a+b, conn };
    let mutex = Mutex::new(my_new_struct);
    let container = Container { mutex };
    ResourceArc::new(container)
}

#[rustler::nif]
fn print(arc: ResourceArc<Container>) {
    let r = arc.mutex.lock().unwrap();
    println!("{:?}, {:?}", r.a, r.conn.type_id());

    let mut statement = r.conn
        .prepare("SELECT 99")
        .unwrap();


    while let State::Row = statement.next().unwrap() {
        println!("result = {}", statement.read::<i64>(0).unwrap());
    }
}

rustler::init!("Elixir.PwTemplate", [add, print], load = load);

and test from elixir:

iex(2)> x = PwTemplate.add(1,2)
#Reference<0.4084525009.3044147210.45533>

and test feedback:

iex(3)> PwTemplate.print(x)    
3, TypeId { t: 4628004102010010283 }
                                    result = 99
                                               {}

Thanx all :blush:

1 Like

7 posts were merged into an existing topic: FunctionClauseError on running mix rustler.new (no function clause matching in OptionParser.validate_swit)

I decided to cheat and pass JSON strings back and forth. I care more about time to working code than how fast it runs. GitHub - benhaney/Jsonrs: Rust powered JSON library for Elixir

I’m a little late to this conversation, but was there any reason why you couldn’t fix the “bug” in the Erlang implementation? Surely, others would have found this limitation to be a hindrance too right? Fixing it in the Erlang implementation may have been less work, and more beneficial to others…

Hey Benjamin!

Yes, there were several reasons:

  1. It’s not a bug. Whoever implemented this feature knew what they were doing when they decided to not allow more data per cell.
  2. There was an endeavor to have a patched version of the ODBC driver in C: GitHub - arcusfelis/eodbc: ODBC backend from OTP + some fixes which I tried for my use case. It kinda worked but was far from stable, so I started fooling around with the code but it lead nowhere. And then I had to make a choice because:
  3. I really, really, really don’t enjoy writing C code. Especially not like in this case where you can easily introduce subtle memory sizing bugs (since you basically wouldn’t know how large the payload might be).

So I used rustler, rust odbc and the mentioned custom bridge for the DB and wrote a custom Ecto wrapper on top of it in the end (which really payed dividents because now it’s trivial to change the driver/adapter code). This setup worked remarkably well and we never had a memory corruption bug in the Rust or the Elixir code and the performance was really good.

All in all, I’m not convinced bringing such change upstream would have been a good idea under the given circumstances and it would absolutely have been much, much more work. At least for me. :smiley: