Arrays - Fast and versatile arrays with swappable implementations

I have been experimenting with implementing an Arrays implementation in native code, by wrapping an immutable datastructure written in Rust using NIFs (Natively Implemented Functions).

The result can be found on the ArraysRRBVector repo, or also as the library of the same name which has been published on Hex.PM (v0.1.0, and it is unlikely to be refined into a stable version).

Unfortunately, “if you want to speed it up, use a NIF” is a statement that does not seems to hold much truth for situations such as these, where you want to implement a datatype on which individual operations are expected to run in the order of microseconds :slightly_frowning_face: . I’ve benchmarked the Rust-based implementation against the existing ErlangArray and MapArray ones, and it turns out that the overhead of the NIF-calls significantly overshadows the otherwise highly performant techniques used in this implementation. (Benchmark graphs can can be found in the README).

So, this was a worthwhile experiment, but it probably makes little sense to use the NIF-based array in production circumstances over the other ones. Maybe there is something I have missed in the implementation, and it still might serve as a nice example of ‘how to implement a NIF-based datastructure’, but it is not the “super efficient” solution that I’d hoped it would be.

So, back to the drawing board it is! Time to look for (or build) one of the more modern array-implementations in plain Erlang or Elixir code.

2 Likes