Rust’s main JSON parser would definitely be the one on serde, that doesn’t mean it’s the fastest (although it is very fast), but it is practically codeless to use it on the Rust side. ^.^
Ooo, nice!
OvermindDL1
ityonemo
Rust’s main JSON parser would definitely be the one on serde, that doesn’t mean it’s the fastest (although it is very fast), but it is practically codeless to use it on the Rust side. ^.^
I take it that means that rustler automatically converts rusts’ map type to erlang’s? so you have to first serialize into rust, and then reserialize into BEAM?
0
dimitarvp
Rustler will automatically serialise most Erlang terms to the Rust code. The reverse is also true. There are exceptions but it’s really easy to specify how should they be serialised as well.
In terms of programmer ergonomics Rustler is quite good (but I do mean the version that’s currently in a release candidate state; the one before it required a good amount of boilerplate for you to write).
So yeah, you’d have Rustler’s lightweight serialisation between the BEAM and the Rust code (on top of serde_json library’s run time). I’d imagine it’d still be miles faster than Jason though – but this should be tested.
There’s a library that seems to do that btw: GitHub - thomas9911/data_serialize: serialize elixir maps to other formats using rust's serde · GitHub. Not sure if the author dwells here on ElixirForum but it should be pretty easy to use it and benchmark against your lib.
1
ityonemo
holy mother of pearl. Zig has generic hashmaps, so I think it would be pretty easy to write a shim with the builtin beam map functions and just plug it in to the std library json parser.
1
OvermindDL1
Oh yeah, I remember that, I think the author has popped up on here actually…
There is also this too, it’s a serde interface for converting to/from ETF’s in rustler if you don’t want to use the normal rustler interface, I think it had a tiny bit of overhead compared to using them raw but I bet that can be fixed:
And don’t forget about the hex library jiffy, it’s a C NIF that binds to some json library to do things there too, it would be a good comparison as well. Since zig can compile C as well I’ve no doubt it could go faster.
Don’t forget to use dirty NIF’s for json processing though!!!
2
dimitarvp
Which kind of them though? Rustler can only mark a function as “dirty I/O” or “dirty CPU” but not both… 
0
ityonemo
Dirty CPU, if you use michal’s benchmarking in Jason. Elixir takes care of the binary loading. I might try doing it as a “long” nif, which uses code that spawns an os thread using the built-in BEAM adapters for that (currently broken in 0.3.0), which has the advantage that it doesn’t consume a dirty nif scheduler, but it might take longer for the OS to do the thread creation.
1
OvermindDL1
Yes, Dirty CPU. Dirty I/O means you won’t be doing much processing so can be evicted. Dirty CPU means you will be consuming at least a good chunk of CPU, regardless of anything else you are doing.
But then you still need to synchronize some other way, it’s far far better to just use a dirty nif scheduler, they are very cheap now in recent OTP versions and doing custom thread handling should only be done in certain extremely specialized cases.
1
ityonemo
But then you still need to synchronize some other way.
If you choose to do a ‘long’ nif, Zigler wraps your function into two nifs, one which launches the process and wraps the frame information into a resource. The second nif retrieves the resource. The first nif’s wrapping function also shoots a message back to the calling process; this is intercepted on the OTP side, which calls the second nif, so indeed the synchronization is taken care of for you.
0
OvermindDL1
Remember calling a NIF is not cheap, you want to minimize the number of calls and do as much work during a single call as is possible. That sounds like it’s doubling the cost?
0