I am thinking of writing a backend for Apache Arrow. I think such a backend would be beneficial as our 2 “backends”, the EXLA compiler and the Torchx backend are ML oriented. Extra overhead of using and installing the above wouldn’t have any benefits if I was just using Nx for data manipulation as opposed to using the binary backend (except for the fact that the binary backend is slower as it uses Beam to hold types)
Thus I feel a general purpose “light” backend that is easy to install and is usable for any serious computing is missing. Apache Arrow is pretty light and fast and fits the bill easily.
So, I am thinking of writing a backend for Arrow. I am going to use the rust arrow2 implementation and write a NIF for it via Rustler ( I think writing an Erlang implementation is a herculean task and would require a NIF anyway for Google framebuffers.)
Any thoughts? Would the Nx community find this useful? Should I work on such a thing?