If anyone is curious about the NIF overhead, here is JLouis’s post about it:
https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7
OvermindDL1
WolfDan
@jc00ke That’s what I’m looking for actually! I hope he can achieve it! (Already exist a library for GraphQL on rust called juniper)
@vonH Oh thanks to share the Nim lang, looks pretty interesting, I also have “My Struggles with Rust”, the assing of variables and mut/ref variables handling and the usize on almost everything is making me crazy hahaha. Nim looks really good
@GregFreeman oh so many thanks, I think thats the missing example that people need, I think can be cool an example on a phoenix umbrella project
@OvermindDL1 You’re my hero, thats what I’m afraid, I was really sure that this interop need some rules to be implemented, with this now I can take more care on this “rules” and write better code
3
jc00ke
GregFreeman
Because I needed to figure it out for my own sake, I also added an example of using Serde and Poison to the phoenix_rustler_integration github example.
This allows you to build up a data structure in Elixir, encode it as JSON and send it to the Rust function, do things with the equivalent structure in Rust with the Serde Macros, and then pass it back into Elixir, which can then decode it with Poison.
This is a LOT easier to do than the alternatives. Probably not quite as fast, but it depends on what your purposes are.
It could probably be changed to BSON or CBOR or what have you.
I’m thinking about it, and I’m not sure if what I just did is crazy or not. But it certainly was a lot simpler than having to manually decode the tuples or maps for any data structures with significant complexity. If you are doing heavy computation on the Rust side (which is why I am using it), and outputting JSON to be handled by the client, then maybe that is the main benefit. You can go DIRECTLY into the rust structure, and DIRECTLY into a suitable data structure for the client.
0
WolfDan
It’s simply amazing your aproach! I was thinking today how to pass a list of data to rust and you solve it for me hahahaha
Yeah maybe the main problem can be performance, but I think the big fault of performance its Poison, I have used it for a lot of json work and still very slow compared with c# (I know its not a correct comparison but after many time with Json.Net your mind keep this performance)
I think your repo its a good star for any that wants join phoenix and rust so many thanks 
1
cmkarlsson
NIFs have other problems too. If you have an expensive NIF in your hot path even dirty schedulers wont help you.
We did lots of rsa_private_decrypt (30% of our requests) and the BEAM did not behave well at all under load. That is to say it behaved a lot like other languages 
Latency was fluctuating, requests were being starved and it was only using 60% server CPU even thought the scheduler usage was 100%. The reason being the schedulers being held up waiting for NIFs who could not yield.
I always thought NIF was the answer to performance problem, however after converting this part of the code to using ports instead our application all of a sudden behaved as it was supposed to, mainly because the erlang schedulers were able to do what they are supposed to.
This made me more cautious of using NIFs for anything remotely long-running (=> 1ms) and rather using ports for those tasks. (and always measure, measure and measure)
1
Qqwy
TypeCheck Core Team
What about using the Erlang External Term format instead of JSON? Integrations for Rust exists, and it would probably be faster to encode/decode than JSON, with less information being lost underway.
0
GregFreeman
@cmkarlsson and @Qqwy:
Interesting points. There is also GitHub - sile/eetf: A Rust implementation of Erlang External Term Format · GitHub
Do either of you know of a clear tutorial of how to set this up on the elixir/mix side to automate the compilation and generation of the rust libaries?
That’s one thing I seem to think are missing in both. There was enough information in Rustler for me to extend what needed to do with Phoenix. I got ports to work last night with some experimentation, but it was a lot of manual calls to rustc and setting flags in cargo.toml. Not automatic like the Rustler integration.
I’m happy to write a tutorial myself, I just need some guidance. I’ve only been using rust for about 3 weeks now, and Elixir only a little bit off and on until very recently.
0
OvermindDL1
Uh, that is definitely very costly and should not be done, not json, not bson, not cbor or anything of the sort. Rustler is a nif, it is designed for speed, thus you should use the NIF protocol, which is the ETF (Erlang Term Format), especially since Rustler already includes all necessary functions for accessing and modifying and working with it. JSON is not a fast protocol, why do you think Poison is often the bottleneck in web processing of super-fast requests. 
Really though, stick with ETF. If you are manually decoding large ETF structures in Rust then you really need to build more helpers for it, or write a macro to auto-generate them.
3
GregFreeman
You’re right. I think that I’ll need to create macros for my structures and
deal with it directly through a port or rustler nif.
I’m going to try to tackle ports and macros to deal with basic structs on
the rust side.
0