I would like to bring Apache Arrow to Elixir and plan to open source this project once I make some meaningful progress.
I have been following this section in the specification and using binary pattern matching to identify messages within an Arrow stream. A message’s metadata is serialized with FlatBuffers and contains among other things the size of the message body, which is used to determine where the message ends in the stream. Unfortunately there is no official implementation of FlatBuffers for Elixir.
I found an unofficial FlatBuffers library but it doesn’t support the
struct data type used in Arrow’s schema files, plus it’s also a fairly inactive project (last commit was in 2017).
I had an idea of calling the FlatBuffers compiler from Elixir to return JSON which I could parse to retrieve the info I need, but this is a hacky solution and I don’t want to be making external calls to compilers when Arrow is supposed to be fast.
Is there any other way I would be able to deserialize this information for use within Elixir?