Serde_rustler - Serde Serializer and Deserializer for your Rust/Rustler NIFs

I learnt some Rust recently as well as I’m also interested in how to get started with NIFs. If you could add some examples, a guide or even a short tutorial, that would be super useful!

serde isn’t just a common (de)serializer for rust, it is overwhelmingly used in the rust ecosystem. ^.^

Very cool creation of this library though, very very nice!

I don’t suppose you could add normal erlang’s term_to_binary/binary_to_term to the encode/decode benchmarks? They are not actually that fast so I’d like to see how they rank among everything as it is?

I’m too lazy to put up a blog post, but I can walk you thru specific files that might help get you started (and someone please correct me if I get some specific detail wrong):

• lib.rs: This is the Rust half of the NIF. Things to note:
  • rustler_export_nifs!: this is the macro that creates the exposed functions to Elixir, where the first parameter specifies the full Elixir module atom name of the Elixir-half of the NIF, and the second defines the a list of exported functions, where SchedulerFlags::DirtyCpu signifies that this function should run in a dirty scheduler.
  • the function signatures of readme and transcode: these are the function signatures of all basic NIFs.
• serde_rustler_tests.ex: This is the Elixir half of the NIF. By default, all functions should return/throw NIF an error b/c the BEAM has yet to replace these functions with the real NIFs (which wont happen until compile time)
• readme.ex uses the NIF created in the previous file to run the readme function from lib.rs and uses a simple doctest to assert it’s correctness. Notice the other modules defined in that file as they define Records and Structs that map directly to types defined in…
• types.rs: these are just some enums and structs to test serialization and deserialization against. Note that they all derive Serialize and Deserialize as those traits define the serialization and deserialization behaviour and are required by serde. Also note the few #[serde(rename = "Elixir...")] annotations - these tell serde to rename these fields or types during serialization to this full name, b/c doing so allows serde to create atoms for those names (b/c those atoms already exist) rather than the default of creating bitstrings; the right atom names are required by Elixir to directly map these types to Records and Structs (instead of tuples and maps).
• serde_rustler_tests_test.exs and test.rs define the actual tests, and lastly
• benchmarks.exs define the Benchee benchmarks jobs that produced the aforementioned results.

Hope that helps!

I can and I will! I’ll update the thread when they’re done.

@svilen forgot to tag you in the previous response.

@sunny-g
Okay, so some feedback on serde_rustler. I am in a situation where I want to replace JSON serialization in a JSONRPC protocol over HTTP with Erlang Terms in a NIF. Unfortunately serde_rustler is not well suited for this very common use case because the terms produced are awkward to work with (tagged tuples etc). Ideally, we need a serializer that is equivalent to serializing Rust as JSON and decoding that same JSON in Elixir and not this strange format. Why? Because most repositories I work with are already doing quite a bit of polishing of the types for JSON serialization. For example for use in JSONRPC calls. When it becomes too heavy to do the JSONRPC over HTTP or a TCP protocol we immediately are tempted to turn to something like Rustler to avoid the network and serialization overhead. Now if the serialization was equivalent to the JSON one I could simply swap out the JSONRPC implementation with Jason.decode to Rustler plus some serializer and be done with it. But unfortunately, this forces me to change all of my code to read the structure. Furthermore, I have situations where the structure is simply passed through my system and should be treated as opaque but now as a result of the serialization scheme I cannot treat it as opaque because the structure has changed from what is expected by the downstream system.

Thanks for all your hard work. Please see this as constructive criticism.

To be honest, I am happy that JSON-style (de)serialization is not the default, because JSON is less expressive (has less different kinds of types) than both Elixir and Rust.

In your specific case, what about doing the JSON deserialization in Rust (using a Rust JSON decoder), and then transforming the resulting Rust structs into the format serde_rustler/Elixir expect?

JSON serialization is precisely what I am trying to avoid because of the overhead of having to look at every byte (versus term construction which doesn’t require looking at all the bytes). The main issue here is that the NIF is here to address performance issues from things like JSON and HTTP and JSONRPC.

Unfortunately, that is half of the use cases I encounter where a Rust NIF is actually useful. Basically the use cases fall into two categories either you are trying to link into a large project or you are doing a simple binding to an external library. But to be honest for the latter case I typically go directly to C for the NIF implementation since the complexity doesn’t justify using Rust in the majority of cases.

I am talking from a pragmatic point of view. I actually am using these tools and I see these shortcomings. The fact is serde_rustler creates more problems for me than it solves.

@edescourtis So to make sure I understand your use case, you have a JSON-RPC server sending and receiving JSON, and you would ideally want serde_rustler to decode JSON into Erlang terms and vice versa?

Unfortunately serde_rustler is not well suited for this very common use case because the terms produced are awkward to work with (tagged tuples etc).

As-is, all the library enables is producing a specific, known-structure-at-compile-time Rust value for a given Elixir term and producing an Elixir-term equivalent for that value, at which point you can use serde attributes to configure exactly what the mapping between your Elixir terms and Rust values should be. A JSON-specific library could be written as a thin wrapper around serde_json, serde_transcode and serde_rustler, similar to what I did here for the JSON tests, but I haven’t gotten around to it yet.

Regardless, can you provide some data structure examples (or preferably code snippets) to illustrate this shortcoming?

JSON serialization is precisely what I am trying to avoid because of the overhead of having to look at every byte (versus term construction which doesn’t require looking at all the bytes). The main issue here is that the NIF is here to address performance issues from things like JSON and HTTP and JSONRPC.

But this library only addresses performance issues for NIFs performing JSON (or any kind of) (de)serialization, so it’s still unclear to me what exactly is not working for you.

Thanks for all your hard work. Please see this as constructive criticism.

Appreciate it, and I totally do but I’d still like to know what I can/should change, so please provide more details

@edescourtis Also, I expect that if you’re seeing more tagged tuples than expected, it’s possibly because where serde_json would normally serialize struct Rgb(u8, u8, u8) as [u8, u8, u8], serde_rustler tries to preserve as much information about the Rust value by default, opting instead to serialize the value as {:Rgb, u8, u8, u8} (both as a tuple and with it’s tag as an atom, aka as a tagged tuple/Record).

I made this choice for newtype structs + variants and tuples + tuple structs + variants deliberately, but am up for discussion to change it.

Shouldn’t be too hard to add an attribute to allow people to define how to serialize something back and forth, either custom serialization or some pre-built ones, such as for struct representations or so?