Serde_rustler - Serde Serializer and Deserializer for your Rust/Rustler NIFs

rust
nif
encoding
serde_rustler
#1

Hi everyone!

After having spent some time with Rust and Elixir, a few weeks ago I set out to learn more about NIFs and benchmarking. There’s a Rust library I use a lot called serde, which essentially defines traits and types for serializing and deserializing between native Rust types and some encoding format like JSON, CBOR, Protocol Buffers, etc. While serde_eetf exists to translate between Rust and the Erlang term format, it still requires that you to convert those binaries into terms.

So instead, I wrapped the Rust NIF library rustler with some Serde traits and out came serde_rustler, which you can use in your Rust NIFs to natively convert your Rust types into Elixir terms and vice versa! It’s also on crates.io and docs.rs, though the documentation is a little sparse.

The encoding and decoding benchmarks look extremely promising (like, insanely, unbelievably promising), and given I haven’t published much in either Rust or Elixir, I’d be really grateful if anyone could point out how to reconfigure the benchmarks or otherwise improve the code, write better tests, or polish the API.

11 Likes
#2

I learnt some Rust recently as well as I’m also interested in how to get started with NIFs. If you could add some examples, a guide or even a short tutorial, that would be super useful!

2 Likes
#3

serde isn’t just a common (de)serializer for rust, it is overwhelmingly used in the rust ecosystem. ^.^

Very cool creation of this library though, very very nice!

I don’t suppose you could add normal erlang’s term_to_binary/binary_to_term to the encode/decode benchmarks? They are not actually that fast so I’d like to see how they rank among everything as it is? :slight_smile:

3 Likes
#4

I’m too lazy to put up a blog post, but I can walk you thru specific files that might help get you started (and someone please correct me if I get some specific detail wrong):

  • lib.rs: This is the Rust half of the NIF. Things to note:
    • rustler_export_nifs!: this is the macro that creates the exposed functions to Elixir, where the first parameter specifies the full Elixir module atom name of the Elixir-half of the NIF, and the second defines the a list of exported functions, where SchedulerFlags::DirtyCpu signifies that this function should run in a dirty scheduler.
    • the function signatures of readme and transcode: these are the function signatures of all basic NIFs.
  • serde_rustler_tests.ex: This is the Elixir half of the NIF. By default, all functions should return/throw NIF an error b/c the BEAM has yet to replace these functions with the real NIFs (which wont happen until compile time)
  • readme.ex uses the NIF created in the previous file to run the readme function from lib.rs and uses a simple doctest to assert it’s correctness. Notice the other modules defined in that file as they define Records and Structs that map directly to types defined in…
  • types.rs: these are just some enums and structs to test serialization and deserialization against. Note that they all derive Serialize and Deserialize as those traits define the serialization and deserialization behaviour and are required by serde. Also note the few #[serde(rename = "Elixir...")] annotations - these tell serde to rename these fields or types during serialization to this full name, b/c doing so allows serde to create atoms for those names (b/c those atoms already exist) rather than the default of creating bitstrings; the right atom names are required by Elixir to directly map these types to Records and Structs (instead of tuples and maps).
  • serde_rustler_tests_test.exs and test.rs define the actual tests, and lastly
  • benchmarks.exs define the Benchee benchmarks jobs that produced the aforementioned results.

Hope that helps!

3 Likes
#5

I can and I will! I’ll update the thread when they’re done.

@svilen forgot to tag you in the previous response.

2 Likes