The Functionality section in the erl_nif docs mention 3 different strategies for long running NIFs: Yielding NIFs, Threaded NIFs and Dirty NIFs. The rustler docs clearly mention how to implement Dirty NIFs. I also found an obscure functionrustler::thread::spawn that handles setting up a non-Erlang thread to execute a closure and returning the result to the calling Erlang process. This addresses Threaded NIFs.
However, I can’t find a Rust-like way to write Yielding NIFs i.e. NIFs that perform work in chunks, yielding back control to the BEAM every millisecond. I found Rustler’s bindings to the C NIF API, so I could just call enif_schedule_nif itself and be done with it. Unfortunately, I can foresee two problems with this approach:
Calling the C API kind of defeats the purpose of writing in Rust
Calling the C API from Rust is probably far more painful than writing in C in the first place
TLDR; How do you implement Yielding NIFs in Rustler?
In order to eliminate any chance of an XY problem, here’s some more context: I am writing an NIF for a heavily async Rust library. I’ve also chosen tokio as the async runtime. I reckon that since I am binding to asynchronous functions, I might as well pass on the benefits to the BEAM’s scheduler.
A yielding NIF seemed like a better fit than a threaded NIF to me for 2 reasons:
Tokio provides robust scheduling and threading, and exposes this in a task-oriented API, making rustler::thread::spawn and enif_thread_* seem redundant
Yielding NIFs are recommended over Threaded and Dirty NIFs by the docs
Since Tokio threads are separate from the BEAM’s, I can have an initial NIF call create a task, and following calls yield very quickly until a result is received.
You can’t find info because yielding NIFs are not supported in Rustler, neither in Zigler. I bet you can do them in C++ and have a middle ground between raw C and memory safety.
This is not a matter of yielding being bad, but a matter of supporting that kind of NIF being hard to implement and be supported by the language used on the other end. Zigler had yielding support until zig dropped their async support.
For heavily async rust code built on top of tokio, I would implement it within a port server. Basically you write a thin GenServer that call out to a managed external process and communicate with it through stdin/stdout. Yes, there will be some overhead in serialization and de-serialization, but I suppose in your problem domain the cost will be amortized, otherwise you won’t need to do the async thingy in the first place.
True, however it is much harder though. With a port server, it will be bog-standard code on either the elixir or the rust side; no need to use any of the erlang C api, all you need to do is to define a simple wire protocol, or pick one that is well supported on both side, like protobuf or something.
I can argue against port server being simpler. With a C node you don’t have an Elixir side, it just works. No need to decide on a wire format because it’s the Erlang term format. All you need to do is use a C library. Why complicate things?
I guess we have different perspectives. Since I am a lousy Rust programmer, I am hesitant to mix c code and Rust code (don’t know how to debug when things go wrong). On the other hand, I can follow a tutorial to write a Tokio based server and pull in a few crates and glue them together with ease. I agree with you that the total lines of code is likely fewer for a C node implementation.
Yeah, I misread “supported” as “supposed”. I don’t know about Zigler, but I do wish the Rustler team wrote a more ergonomic wrapper around enif_schedule_nif. That way I can handle the exact mechanics of how and when to yield without the Rustler team making too many assumptions on my side. Maybe I should file an issue.
That’s a very interesting approach which I hadn’t considered. In my case I’m writing database drivers, so the cost of serde is non-trivial.
Like Derek mentioned, this is subjective, but I agree with Derek on this. Being primarily an Elixir/Erlang programmer, anything that makes the Rust/C side simpler is better in my book. In the case of a C-node in Rust, I would have to write a Rust wrapper over the ErlInterface API, which is definitely something I don’t want to get into.
Still, C-nodes seem like something worth investigating with potential applications later!
