Benjamin-Philip
The Functionality section in the erl_nif docs mention 3 different strategies for long running NIFs: Yielding NIFs, Threaded NIFs and Dirty NIFs. The rustler docs clearly mention how to implement Dirty NIFs. I also found an obscure function rustler::thread::spawn that handles setting up a non-Erlang thread to execute a closure and returning the result to the calling Erlang process. This addresses Threaded NIFs.
However, I can’t find a Rust-like way to write Yielding NIFs i.e. NIFs that perform work in chunks, yielding back control to the BEAM every millisecond. I found Rustler’s bindings to the C NIF API, so I could just call enif_schedule_nif itself and be done with it. Unfortunately, I can foresee two problems with this approach:
- Calling the C API kind of defeats the purpose of writing in Rust
- Calling the C API from Rust is probably far more painful than writing in C in the first place
TLDR; How do you implement Yielding NIFs in Rustler?
In order to eliminate any chance of an XY problem, here’s some more context: I am writing an NIF for a heavily async Rust library. I’ve also chosen tokio as the async runtime. I reckon that since I am binding to asynchronous functions, I might as well pass on the benefits to the BEAM’s scheduler.
A yielding NIF seemed like a better fit than a threaded NIF to me for 2 reasons:
- Tokio provides robust scheduling and threading, and exposes this in a task-oriented API, making
rustler::thread::spawnandenif_thread_*seem redundant - Yielding NIFs are recommended over Threaded and Dirty NIFs by the docs
Since Tokio threads are separate from the BEAM’s, I can have an initial NIF call create a task, and following calls yield very quickly until a result is received.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
Benjamin-Philip
This post was crossposted to ErlangForums.
krasenyp
You can’t find info because yielding NIFs are not supported in Rustler, neither in Zigler. I bet you can do them in C++ and have a middle ground between raw C and memory safety.
Benjamin-Philip
Why aren’t you supposed to yield in Rustler or Zigler?
I finally went with a Threading NIF as-per the suggestions in the Erlang Forums post.
LostKobrakai
This is not a matter of yielding being bad, but a matter of supporting that kind of NIF being hard to implement and be supported by the language used on the other end. Zigler had yielding support until zig dropped their async support.
derek-zhou
For heavily async rust code built on top of tokio, I would implement it within a port server. Basically you write a thin GenServer that call out to a managed external process and communicate with it through stdin/stdout. Yes, there will be some overhead in serialization and de-serialization, but I suppose in your problem domain the cost will be amortized, otherwise you won’t need to do the async thingy in the first place.
krasenyp
A C node might be a better solution. At least you have all the power of file descriptors and don’t rely on stdin/stdout.
derek-zhou
True, however it is much harder though. With a port server, it will be bog-standard code on either the elixir or the rust side; no need to use any of the erlang C api, all you need to do is to define a simple wire protocol, or pick one that is well supported on both side, like protobuf or something.
krasenyp
I can argue against port server being simpler. With a C node you don’t have an Elixir side, it just works. No need to decide on a wire format because it’s the Erlang term format. All you need to do is use a C library. Why complicate things?
derek-zhou
I guess we have different perspectives. Since I am a lousy Rust programmer, I am hesitant to mix c code and Rust code (don’t know how to debug when things go wrong). On the other hand, I can follow a tutorial to write a Tokio based server and pull in a few crates and glue them together with ease. I agree with you that the total lines of code is likely fewer for a C node implementation.
Benjamin-Philip
Yeah, I misread “supported” as “supposed”. I don’t know about Zigler, but I do wish the Rustler team wrote a more ergonomic wrapper around
enif_schedule_nif. That way I can handle the exact mechanics of how and when to yield without the Rustler team making too many assumptions on my side. Maybe I should file an issue.