thanos

thanos

ExArrow - Apache Arrow IPC, Flight & ADBC Support

ExArrow – Apache Arrow IPC, Flight & ADBC Support for Elixir

I’m announcing the public release of ExArrow:

ExArrow provides a focused, BEAM-friendly implementation of:

  • Arrow IPC encoding / decoding
  • Arrow Flight client support
  • Arrow Database Connectivity (ADBC) integration
  • Explorer interoperability

The library concentrates on Arrow’s memory model and transport protocols.


Design Scope

ExArrow is not:

  • A dataframe library
  • A database driver
  • A replacement for Explorer or ADBC

It is the Arrow-native infrastructure layer.

It exposes Arrow in a way that integrates cleanly with existing Elixir data tooling.


Ecosystem Positioning

Layer Library
Dataframe computation Explorer
Database connectivity ADBC
Chunked storage ExZarr
Columnar transport & memory ExArrow

ExArrow complements these libraries by handling Arrow IPC, Flight, and columnar structures directly.


How It Fits Into Real Workflows

Typical architecture:

  1. ADBC retrieves Arrow data from a database
  2. ExArrow handles IPC or Flight transport
  3. Explorer consumes Arrow-backed dataframes
  4. ExZarr stores large chunked datasets when persistence is required

This layered model enables:

  • Zero-copy data interchange
  • Clean separation of responsibilities
  • Cross-language interoperability
  • Arrow-native services in Phoenix or distributed BEAM systems

Practical Impact

Apache Arrow has become a core interoperability layer in modern analytics systems.

ExArrow makes Arrow directly usable in Elixir without requiring external bridges or conversion layers.

The intent is to provide a stable foundation for columnar-native applications on the BEAM.


Contributions, benchmarks, and production feedback are welcome.

— Thanos Vassilakis


https://github.com/thanos/ex_arrow

First Post!

filmor

filmor

Could you please check the docs at resource_impl in rustler - Rust and implement the resources accordingly? The way you are doing it right now hooks into internal types and is needlessly complicated.

Last Post!

Where Next?

Popular in Announcing Top

Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Thank...
New
Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
mspanc
I am pleased to announce an initial release of the Membrane Framework - an Elixir-based framework with special focus on processing multim...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44532 311
New
RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 12888 134
New
wojtekmach
Hey everyone! Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out...
New
zachdaniel
Ash Framework What is Ash? Ash Framework is a declarative, resource-oriented application development framework for Elixir. A resource can...
New

Other popular topics Top

openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42533 114
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement