billylanchantin

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

The Explorer library has been out for some time. But we just released the latest version (see below), and we thought we’d start posting updates to the forum.

Wait, Explorer is new to me. What are Series and DataFrames?

Explorer is a DataFrame library for Elixir.

DataFrame libraries are common in languages which have a focus on data manipulation, including:

dplyr (R)
Polars (Rust/Python)

If you’d like a more in-depth tutorial, there’s an excellent LiveBook called Ten Minutes to Explorer that you can play with:

Ten Minutes to Explorer — Explorer v0.11.1

But we’ll provide a quick overview here.

Briefly, you can think of a DataFrame like an in-memory table. Its purpose is to facilitate common data exploration and analysis tasks. As such, it’s a column-oriented table.

Column-oriented tables

If you’re unfamiliar with column-oriented tables, suppose you have a table of pet data like this:

type	age	color
cat	5	black
dog	2	brown
dog	3	brindle

A row-oriented organization of that data might look like this in Elixir:

rows = [
  [type: "cat", age: 5, color: "black"],
  [type: "dog", age: 2, color: "brown"],
  [type: "dog", age: 3, color: "brindle"],
]

It matches the original table fairly one-to-one. But the column-oriented version might instead look like:

columns = [
  type: ["cat", "dog", "dog"],
  age: [5, 2, 3],
  color: ["black", "brown", "brindle"]
]

It has same information, but “transposed”.

Column-orientation is beneficial if you’re asking questions that require a lot of number-crunching like “What’s the average age of all pets?”. In the row-oriented version, finding the average age would require first looking through the entire contents of the table to collect the relevant data. But in the column-oriented version, those values have already been co-located in memory.

Series and DataFrames: columns and tables

In dataframe parlance, a “series” is a single column and a “dataframe” is a collection of named series, aka a table.

Our example above would look like this:

type = Explorer.Series.from_list(["cat", "dog", "dog"])
age = Explorer.Series.from_list([5, 2, 3])
color = Explorer.Series.from_list(["black", "brown", "brindle"])

df = Explorer.DataFrame.new(type: type, age: age, color: color)
# #Explorer.DataFrame<
#   Polars[3 x 3]
#   type string ["cat", "dog", "dog"]
#   age s64 [5, 2, 3]
#   color string ["black", "brown", "brindle"]
# >

Some things to note:

Each series has a corresponding data type or “dtype”, e.g. type has the dtype string.
The word “Polars” appears. That indicates that this dataframe is using the backend powered by the fantastic Polars library (the default backend).

And if we really did want to know the average age of the pets, that would look like this:

Explorer.Series.mean(df["age"])
# 3.3333333333333335

Features and design

Preiminaries out of the way, here are Explorer’s high-level features:

Simply typed series: :binary, :boolean, :category, :date, :datetime, :duration, floats of 32 and 64 bits ({:f, size}), integers of 8, 16, 32 and 64 bits ({:s, size}, {:u, size}), :null, :string, :time, :list, and :struct.
A powerful but constrained and opinionated API, so you spend less time looking for the right function and more time doing data manipulation.
Support for CSV, Parquet, NDJSON, and Arrow IPC formats
Integration with external databases via ADBC and direct connection to file storages such as S3
Pluggable backends, providing a uniform API whether you’re working in-memory or (forthcoming) on remote databases or even Spark dataframes.
The first (and default) backend is based on NIF bindings to the blazing-fast polars library.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr.
The philosophy is heavily influenced by this passage from dplyr’s documentation:

By constraining your options, it helps you think about your data manipulation challenges.

It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate your thoughts into code.

It uses efficient backends, so you spend less time waiting for the computer.

The aim here isn’t to have the fastest dataframe library around (though it certainly helps that we’re building on Polars, one of the fastest).
Instead, we’re aiming to bridge the best of many worlds:

the elegance of dplyr
the speed of polars
the joy of Elixir

That means you can expect the guiding principles to be ‘Elixir-ish’. For example, you won’t see the underlying data mutated, even if that’s the most efficient implementation. Explorer functions will always return a new dataframe or series.

Acknowledgements

Explorer is an extensive library and there’s much more we could say. But for now, we’d just like to thank the dozens of contributors who’ve added wonderful improvements over the years.

16 comments

#library #explorer

21 1986 16

2025-08-30 15:21:14 UTC

First 10 of 16 Posts

billylanchantin

Explorer - version 0.8

Explorer has released version 0.8!

Added

Add explode/2 to Explorer.DataFrame. This function is useful to expand the contents of a {:list, inner_dtype} series into a “inner_dtype” series.
Add the new series functions all?/1 and any?/1, to work with boolean series.
Add support for the “struct” dtype. This new dtype represents the struct dtype from Polars/Arrow.
Add map/2 and map_with/2 to the Explorer.Series module.
This change enables the usage of the Explore.Query features in a series.
Add sort_by/2 and sort_with/2 to the Explorer.Series module.
This change enables the usage of the lazy computations and the Explorer.Query module.
Add unnest/2 to Explorer.DataFrame. It works by taking the fields of a “struct” - the new dtype - and transform them into columns.
Add pairwise correlation - Explorer.DataFrame.correlation/2 - to calculate the correlation between numeric columns inside a data frame.
Add pairwise covariance - Explorer.DataFrame.covariance/2 - to calculate the covariance between numeric columns inside a data frame.
Add support for more integer dtypes. This change introduces new signed and unsigned integer dtypes:
- {:s, 8}, {:s, 16}, {:s, 32}
- {:u, 8}, {:u, 16}, {:u, 32}, {:u, 64}.
The existing :integer dtype is now represented as {:s, 64}, and it’s still the default dtype for integers. But series and data frames can now work with the new dtypes. Short names for these new dtypes can be used in functions like Explorer.Series.from_list/2. For example, {:u, 32} can be represented with the atom :u32.

This may bring more interoperability with Nx, and with Arrow related things, like ADBC and Parquet.
Add ewm_standard_deviation/2 and ewm_variance/2 to Explorer.Series.
They calculate the “exponentially weighted moving” variance and standard deviation.
Add support for :skip_rows_after_header option for the CSV reader functions.
Support {:list, numeric_dtype} for Explorer.Series.frequencies/1.
Support pins in cond, inside the context of Explorer.Query.
Introduce the :null dtype. This is a special dtype from Polars and Apache Arrow to represent “all null” series.
Add Explorer.DataFrame.transpose/2 to transpose a data frame.

Changed

Rename the functions related to sorting/arranging of the Explorer.DataFrame.
Now arrange_with is named sort_with, and arrange is sort_by.

The sort_by/3 is a macro and it is going to work using the Explorer.Query module. On the other side, the sort_with/2 uses a callback function.
Remove unnecessary casts to {:s, 64} now that we support more integer dtypes.
It affects some functions, like the following in the Explorer.Series module:
- argsort
- count
- rank
- day_of_week, day_of_year, week_of_year, month, year, hour, minute, second
- abs
- clip
- lengths
- slice
- n_distinct
- frequencies
And also some functions from the Explorer.DataFrame module:
- mutate - mostly because of series changes
- summarise - mostly because of series changes
- slice

Fixed

Fix inspection of series and data frames between nodes.
Fix cast of :string series to {:datetime, any()}
Fix mismatched types in Explorer.Series.pow/2, making it more consistent.
Normalize sorting options.
Fix functions with dtype mismatching the result from Polars.
This fix is affecting the following functions:
- quantile/2 in the context of a lazy series
- mode/1 inside a summarisation
- strftime/2 in the context of a lazy series
- mutate_with/2 when creating a column from a NaiveDateTime or Explorer.Duration.

Contributors

Thank you to everyone who opend up a PR:

@billylanchantin
@cigrainger
@costaraphael
@cristineguadelupe
@Jhonatannunessilva
@kellyfelkins
@lkarthee
@philss

And thank you to the first-time contributors!:

@JonGretar made their first contribution in Support for skip_rows_after_header option in reading csv files by JonGretar · Pull Request #782 · elixir-explorer/explorer · GitHub
@rtvu made their first contribution in Added Series.ewm_std/2 and Series.ewm_var/2 by rtvu · Pull Request #778 · elixir-explorer/explorer · GitHub

Changelogs

Full Changelog: Comparing v0.7.2...v0.8.0 · elixir-explorer/explorer · GitHub
Official Changelog: Changelog — Explorer v0.11.1

Post #1

billylanchantin

[Blog] Explorer 0.8: The `dtype` release

https://cigrainger.com/explorer-0-8-0/

Post #2

billylanchantin

[Release] Explorer - v0.9

Added

Add initial support for SQL queries.

The Explorer.DataFrame.sql/3 is a function that accepts a dataframe and a SQL query. The SQL is not validated by Explorer, so the queries will be backend dependent. Right now we have only Polars as the backend.
Add support for remote series and dataframes.

Automatically transfer data between nodes for remote series and dataframes and perform distributed garbage collection.

The functions in Explorer.DataFrame and Explorer.Series will automatically move operations on remote dataframes to the nodes they belong to.
The Explorer.Remote module provides additional conveniences for manual placement.
Add FLAME integration, so we automatically track remote series and dataframes returned from FLAME calls when the :track_resources option is enabled.
See FLAME for more.
Add Explorer.DataFrame.transform/3 that applies an Elixir function to each row. This function is similar to Explorer.Series.transform/2, and as such, it’s considered an expensive operation. So it’s recommended only if there is no similar dataframe or series operation available.
Improve performance of Explorer.Series.from_list/2 for most of the cases where the :dtype option is given. This is specially true for when the dtype is :binary.

Changed

Stop inference of dtypes if the :dtype option is given by the user.
The main goal of this change is to improve performance. We are now delegating the job of decoding the terms as the given :dtype to the backend.
Explorer.Series.pow/2 no longer casts to float when the exponent is a signed integer. We are following the way Polars works now, which is to try to execute the operation or raise an exception in case the exponent is negative.
Explorer.Series.pivot_wider/4 no longer includes the names_from column name in the new columns when values_from is a list of columns. This is more consistent with its behaviour when values_from is a single column.
Explorer.Series.substring/3 no longer cycles to the end of the string if the negative offset surpasses the beginning of that string. In that case, an empty string is returned.
The Explorer.Series.ewm_* functions no longer replace nil values with the value at the previous index. They now propogate nil values through to the result series.
Saving a dataframe as a Parquet file to S3 services no longer works when streaming is enabled. This is temporary due to a bug in Polars. An exception should be raised instead.

Contributors

And thank you to the first-time contributors!:

@elbow-jason made their first contribution in Fix DateTime -> ExDateTime lifetime issue by elbow-jason · Pull Request #909 · elixir-explorer/explorer · GitHub
@zhzy0077 made their first contribution in Docs: replace bad link for nsjson by zhzy0077 · Pull Request #940 · elixir-explorer/explorer · GitHub

Changelogs

Full changelog: Comparing v0.8.3...v0.9.0 · elixir-explorer/explorer · GitHub
Official changelog: Changelog — Explorer v0.11.1

Post #3

billylanchantin

[Release] Explorer - v0.10

Added

Add support for the decimals data type.

Decimals dtypes are represented by the {:decimal, precision, scale} tuple,
where precision can be a positive integer from 0 to 38, and is the maximum number
of digits that can be represented by the decimal. The scale is the number of
digits after the decimal point.

With this addition, we also added the :decimal package as a new dependency.
The Explorer.Series.from_list/2 function accepts decimal numbers from that
package as values * %Decimal{}.

This version has a small number of operations, but is a good foundation.
Allow the usage of queries and lazy series outside callbacks and macros.
This is an improvement to functions that were originally designed to accept callbacks.
With this change you can now reuse lazy series across different “queries”.
See the Explorer.Query docs for details.

The affected functions are:
- Explorer.DataFrame.filter_with/2
- Explorer.DataFrame.mutate_with/2
- Explorer.DataFrame.sort_with/2
- Explorer.DataFrame.summarise_with/2
Allow accessing the dataframe inside query.
Add “lazy read” support for Parquet and NDJSON from HTTP(s).
Expose more options for Explorer.Series.cut/3 and Explorer.Series.qcut/3.
These options were available in Polars, but not in our APIs.

Fixed

Fix creation of series where a nil value inside a list * for a {:list, any()} dtype -
could result in an incompatible dtype. This fix will prevent panics for list of lists with
nil entries.
Fix Explorer.DataFrame.dump_ndjson/2 when date time is in use.
Fix Explorer.Series.product/1 for lazy series.
Accept %FSS.HTTP.Entry{} structs in functions like Explorer.DataFrame.from_parquet/2.
Fix encode of binaries to terms from series of the {:struct, any()} dtype.
In case the inner fields of the struct had any binary (:binary dtype), it was
causing a panic.

Changed

Change the defaults of the functions Explorer.Series.cut/3 and Explorer.Series.qcut/3
to not have “break points” column in the resultant dataframe.
So the :include_breaks is now false by default.

Contributors

New Contributors

@audacioustux made their first contribution in fix typo cuontry ~> country by audacioustux · Pull Request #979 · elixir-explorer/explorer · GitHub
@ceyhunkerti made their first contribution in LazyFrame from http endpoint by ceyhunkerti · Pull Request #993 · elixir-explorer/explorer · GitHub
@mooreryan made their first contribution in Fix return type in `DataFrame.dtypes/1` typespec (#1002) by mooreryan · Pull Request #1003 · elixir-explorer/explorer · GitHub
@preciz made their first contribution in Fix typos by preciz · Pull Request #984 · elixir-explorer/explorer · GitHub

Changelogs

Full changelog: Comparing v0.9.2...v0.10.0 · elixir-explorer/explorer · GitHub
Official changelog: Changelog — Explorer v0.11.1

Post #4

tmbb

I have a question regarding the development of new “primitive” functions.

I want to write functions that sample vectors (series?) of values from probability distributions. I’m most interested in the Beta and Dirichlet distributions.

I’ve thought of a format that handles both univariate (e.g. Beta) and multivariate (e.g. Dirichlet) distributions.

For univariate distributions it would be something like this:

DataFrame.draw_from_beta(a, b, n) = DataFrame.new(draws: [1,2,3,..., n], values: [...])

For multivariate distributions it would be something like this:

# For length(alpha) == 4
DataFrame.draw_from_dirichlet(alphas, n) =
  DataFrame.new(
    draws: [1,1,1,1,1,2,2,2,2, ... , n,n,n,n], # length = 4 * n
    i: [1,2,3,4, 1,2,3,4, ...], # length = 4 * n
    values: [...] # length = 4 * n
   )

Using a dataframe instead of a series allows me to have more or less uniform encoding for both univarite and multivariate distributions that plays well with the Explorer functions.

In order to implement this efficiently, I am building the dataframe fully in rust using a rust function and binding it to the Native module.

My questions are:

Is there any way of implementing this outside of Explorer? I believe there isn’t, because the Rust types are defined inside Explorer’s source tree
Is there any way the base types could be split into a rust crate so that they could be used in different elixir packages? I understand it’s probably more work than it’s worth
Would Explorer accept the inclusion of these functions as part of a larger group of statistical functions (random variable sampling, CDF and PDF evaluation, etc.)
If so, should those functions live in the Explorer.DataFrame module (because they generate dataframes) or in a new Explorer.Distribution module or something (like Explorer.Math or Explorer.Statistics)

Some final thoughts: I wonder if dataframes are the right API for this, but the truth is that Elixir doesn’t have bindings to a good array library, and even if it did, representing multidimensional arrays as a columns in a dataframe indexed by multiple columns doesn’t seem that terrible, and it might lead to an API which is actually better than some messy things I’ve had to do with NumPy arrays

Post #5

josevalim

Creator of Elixir

I wonder if you could implement these functions in Nx? This means you get to implement them in Elixir and they should be highly efficient as well.

Although we discussed ideas for exposing an Elixir API that would allow you to pass user defined functions, implemented in Rust, to Polars, so you can extend them.

Post #6

tmbb

I wonder if you could implement these functions in Nx?

I’ve just checked and Nx does provide a function to generate uniform random number between 0 and 1, so yeah, I could probably sample from whatever distributions I wanted. However I have never actually been able to make Nx work with the EXLA compiler (or with any other compiler backend, IIRC). I didn’t actually try that hard, though.

I can try to make it work again, I guess…

Although we discussed ideas for exposing an Elixir API that would allow you to pass user defined functions, implemented in Rust, to Polars, so you can extend them.

How would that work? Would you use the C-ABI to interface between dynamically linked rust libraries from different Elixir packages?

Post #7

billylanchantin

How would [exposing an Elixir API that would allow you to pass user defined functions] work?

Polars has a plugin feature. IIRC the plan was to try and piggyback off that. But we’d not worked out the details.

I’ll also echo that this sounds more like a case for Nx.Tensors. I would start by trying to implement draw_from_beta, draw_from_dirchlet, etc. as defns as that seems like the more natural setting to me.

Post #8

tmbb

Hm… Maybe, I don’t know… Drawing random numbers and putting them in a dataframes for further analysis (let’s say those numbers are posterior samples of a parameter in the context of a Bayesian analysis) seems like quite a natural thing to do.

But I’ll give Nx a try, anyway

Post #9

tmbb

I still can’t get the EXLA compiler to run on my system (64bit WSL running on Windows), so I went ahead and implemented the random distributions in a fork. I’ll polish it a bit and make a case for inclusion in Explorer.

I think Explorer should also support KDE for random samples, with decent bandwidth estimations. I have successfully implemented KDEs using the Silverman’s rule for bandwidth estimation in Elixir (on top of Explorer, of course), and it’s quite performant, but it’s important to support better methods like the improved Sheather-Jones method, which definitely needs to be implemented in “raw” rust with access to sophisticated rust packages.

Having good support for KDEs (or for binning in histograms) is important for plotting random distributions.

Post #10

Load more posts