Backend for Apache Arrow

I am thinking of writing a backend for Apache Arrow. I think such a backend would be beneficial as our 2 “backends”, the EXLA compiler and the Torchx backend are ML oriented. Extra overhead of using and installing the above wouldn’t have any benefits if I was just using Nx for data manipulation as opposed to using the binary backend (except for the fact that the binary backend is slower as it uses Beam to hold types)

Thus I feel a general purpose “light” backend that is easy to install and is usable for any serious computing is missing. Apache Arrow is pretty light and fast and fits the bill easily.

So, I am thinking of writing a backend for Arrow. I am going to use the rust arrow2 implementation and write a NIF for it via Rustler ( I think writing an Erlang implementation is a herculean task and would require a NIF anyway for Google framebuffers.)

Any thoughts? Would the Nx community find this useful? Should I work on such a thing?

The Explorer project uses Polars which in turn uses Apache Arrow. IIUC Arrow is more for data analytics and manipulation than for the applications Nx is intended for. For example, I’m not really sure it’d be possible to implement Nx.conv using Apache Arrow (just one example).

If you are interested in writing an Nx backend or compiler, there are many options out there that could be fun to explore:

These are just to name a few :slight_smile:

1 Like