What do you think about integrating Rust and Elixir?

After a lot of struggle and seemingly trying everything possible to get ports working, including trying the difficult port drivers and digging into erlang calls themselves, I got a rust/elixir port working!

The embarrassing part was I didn’t realize that a newline character was needed for the rust read_line function to return.

However, this is exactly what I’ve been after! A great language for a distributed webserver and a great language for nitty gritty numerical work that isn’t as unsafe as C or as archaic as Fortran. Then, add Phoenix, throw in some JS/HTML/CSS/SQL for good measure, and you have everything you could possibly want.

Example posted here, with the Nif example: GitHub - cgregfreeman/phoenix_rust_ports_and_nifs: Examples of Phoenix with Rust interfaces via both ports and nifs · GitHub

Now I can get on with what I originally set out to do.

Thank you everyone here. I hope my examples are useful to you.

Thanks,

That actually told me quite a bit. In trying to figure out Ports, I got bare ports to work with binary files. However, the trouble came up when trying to get the standard compilation to dylibs by Rustler. I could only get port drivers to work in that case.

However, by playing around with the cargo.toml for the port code, I was able to get the binary to work. It’s automatically recompiled with rustler, but the binary file doesn’t get placed in the proper native folder. So, it’s a partial solution.

I couldn’t figure out the rustler mix task that may allow for binary compilation…maybe the owner will be nice and either tell me how I should do it or someone else knows.

Hi Greg,

Sorry if this is a bit of a non-answer to the actual question.

You can use linked in drivers but I think you want a plain port. Port Drivers existed before NIFs and also has the drawback of crashing the entire VM if something goes wrong. I think most people use NIFs instead of Port Drivers today but I may be wrong. For example I know the crypto library in erlang used to be a port driver but is now implemented as a NIF.

A plain port is just a standalone application which communicate with erlang using file descriptors (they recommend 3 and 4 but stdin/stdout can be used to)

All the integration options are found here:

And the normal port integration here:

The port example uses C but any language can be used. There are no dependencies on erlang at all. The only thing is that you have to implement your custom protocol to talk to the port.

Here’s the state of my journey with ports with Rust and Elixir:

I can get Nifs to work fairly well, however I understand the disadvantages.

However, getting ports to work has been a bit of a nightmare because help is sparse.

I finally got to the point where I think I’m close, but not there yet.

Using Rustler, I can get the linking to work and point to the .so (driver) compiled file.

I can then attempt to load the driver using :erl_ddll.load(“path/to/so”, :soname). But, I get the error from erlang that “No driver init in dynamic library.”

Is there anyone out there that can help me out with fixing that?

Ok it looks like I’ll still need to figure out something to manage automatic compilation with Mix. EETF is purely Rust code.

I could take the existing Rustler Mix code, as linked above, and re-purpose it to manage the Rust files.

I think that’s what I’m really asking. Rustler is really nice in that it can identify if compilation is needed and sets up taking care of the paths and binaries.

EETF by itself is not enough to make a project tying both Elixir and Rustl together very automatic. This is probably why Rustler is so much more popular.

eetf is all you need to do rust Ports. A port is just an external running application communication with the EVM via stdin/stdout, that is all, and if you want to use erlang’s external term format for speed then eetf handles that, or you can marshall information however you want otherwise.

@cmkarlsson and @OvermindDL1

Are you aware of any open source code that takes care of the Rust compilation and path management for ports?

I’ve been taking a look at https://github.com/hansihe/rustler/blob/master/rustler_mix/lib/mix/tasks/compile.rustler.ex
and the rest of the Rustler project code and managing ports on a project needs something similar, if I’m not mistaken.

I’ve thought about branching those parts of Rustler needed for ports but I don’t want to repeat it if it’s already been done somewhere else, or if it’s actually not the right thing to do. I’ve also seen the comments here: Feature request: support for ports · Issue #78 · rusterlium/rustler · GitHub

It doesn’t look that trivial at all to set up a Rust compiler in a mix project, unless I’m missing something.

You’re right. I think that I’ll need to create macros for my structures and
deal with it directly through a port or rustler nif.

I’m going to try to tackle ports and macros to deal with basic structs on
the rust side.

Uh, that is definitely very costly and should not be done, not json, not bson, not cbor or anything of the sort. Rustler is a nif, it is designed for speed, thus you should use the NIF protocol, which is the ETF (Erlang Term Format), especially since Rustler already includes all necessary functions for accessing and modifying and working with it. JSON is not a fast protocol, why do you think Poison is often the bottleneck in web processing of super-fast requests.

Really though, stick with ETF. If you are manually decoding large ETF structures in Rust then you really need to build more helpers for it, or write a macro to auto-generate them.

@cmkarlsson and @Qqwy:

Interesting points. There is also GitHub - sile/eetf: A Rust implementation of Erlang External Term Format · GitHub
Do either of you know of a clear tutorial of how to set this up on the elixir/mix side to automate the compilation and generation of the rust libaries?

That’s one thing I seem to think are missing in both. There was enough information in Rustler for me to extend what needed to do with Phoenix. I got ports to work last night with some experimentation, but it was a lot of manual calls to rustc and setting flags in cargo.toml. Not automatic like the Rustler integration.

I’m happy to write a tutorial myself, I just need some guidance. I’ve only been using rust for about 3 weeks now, and Elixir only a little bit off and on until very recently.