Apache Arrow

Following. Going to be building parquet files via Broadway soon also.

I see Elixir and Rust as brothers that are complementing each other and I’d love to bring some performance-critical (and useful!) pieces of technology to the Elixir ecosystem through Rust NIFs.

(When I settle in my new job – should be soon – I plan to finish my sqlite driver + Ecto 3.x adapter which uses a Rust library beneath.)

I have worked for some time with Rustler (0.22-rc) and I quite liked it. There are a few things it still needs to do better – like give you an explicit API for yielding back to the BEAM so you can manually make sure you are not clogging the dirty NIF schedulers (scroll down and look for “Long-running NIFs” for context) – but in general it has been an absolutely excellent project! (@scrogson do let me know if I am misrepresenting Rustler, has been a few months since I last looked at it.)

You only need to make sure to periodically yield to the BEAM VM with chunks of the work being done and you’ll have one lag-less and lighting-fast program.

To me Elixir and Rust pair naturally because Rust can make certain mutable data structures work extremely fast and Elixir can literally do everything else (including orchestration between services).

I might be overly optimistic here but I can see a future where Elixir and Rust converge together in a co-op environment / runtime, f.ex. the Lumen effort.

I don’t think this is overly optimistic necessarily. Hans (hansihe) works on both Lumen and Rustler. I know the Lumen folks are hoping to be compatible with existing C NIFs to not lose large amounts of the ecosystem. But I also think there is a great value in strengthening the Elixir + Rust case. Lumen is built with Rust. I guess we might stick with the Rustler approach though to not cut ties with C land. I think that would be a mistake.