News>Announcing

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

#library#explorer
billylanchantin

billylanchantin

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

The Explorer library has been out for some time. But we just released the latest version (see below), and we thought we’d start posting updates to the forum.

Wait, Explorer is new to me. What are Series and DataFrames?

Explorer is a DataFrame library for Elixir.

DataFrame libraries are common in languages which have a focus on data manipulation, including:

If you’d like a more in-depth tutorial, there’s an excellent LiveBook called Ten Minutes to Explorer that you can play with:

But we’ll provide a quick overview here.

Briefly, you can think of a DataFrame like an in-memory table. Its purpose is to facilitate common data exploration and analysis tasks. As such, it’s a column-oriented table.

Column-oriented tables

If you’re unfamiliar with column-oriented tables, suppose you have a table of pet data like this:

type age color
cat 5 black
dog 2 brown
dog 3 brindle

A row-oriented organization of that data might look like this in Elixir:

rows = [
  [type: "cat", age: 5, color: "black"],
  [type: "dog", age: 2, color: "brown"],
  [type: "dog", age: 3, color: "brindle"],
]

It matches the original table fairly one-to-one. But the column-oriented version might instead look like:

columns = [
  type: ["cat", "dog", "dog"],
  age: [5, 2, 3],
  color: ["black", "brown", "brindle"]
]

It has same information, but “transposed”.

Column-orientation is beneficial if you’re asking questions that require a lot of number-crunching like “What’s the average age of all pets?”. In the row-oriented version, finding the average age would require first looking through the entire contents of the table to collect the relevant data. But in the column-oriented version, those values have already been co-located in memory.

Series and DataFrames: columns and tables

In dataframe parlance, a “series” is a single column and a “dataframe” is a collection of named series, aka a table.

Our example above would look like this:

type = Explorer.Series.from_list(["cat", "dog", "dog"])
age = Explorer.Series.from_list([5, 2, 3])
color = Explorer.Series.from_list(["black", "brown", "brindle"])

df = Explorer.DataFrame.new(type: type, age: age, color: color)
# #Explorer.DataFrame<
#   Polars[3 x 3]
#   type string ["cat", "dog", "dog"]
#   age s64 [5, 2, 3]
#   color string ["black", "brown", "brindle"]
# >

Some things to note:

  • Each series has a corresponding data type or “dtype”, e.g. type has the dtype string.
  • The word “Polars” appears. That indicates that this dataframe is using the backend powered by the fantastic Polars library (the default backend).

And if we really did want to know the average age of the pets, that would look like this:

Explorer.Series.mean(df["age"])
# 3.3333333333333335

Features and design

Preiminaries out of the way, here are Explorer’s high-level features:

  • Simply typed series: :binary, :boolean, :category, :date, :datetime, :duration, floats of 32 and 64 bits ({:f, size}), integers of 8, 16, 32 and 64 bits ({:s, size}, {:u, size}), :null, :string, :time, :list, and :struct.

  • A powerful but constrained and opinionated API, so you spend less time looking for the right function and more time doing data manipulation.

  • Support for CSV, Parquet, NDJSON, and Arrow IPC formats

  • Integration with external databases via ADBC and direct connection to file storages such as S3

  • Pluggable backends, providing a uniform API whether you’re working in-memory or (forthcoming) on remote databases or even Spark dataframes.

  • The first (and default) backend is based on NIF bindings to the blazing-fast polars library.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr.
The philosophy is heavily influenced by this passage from dplyr’s documentation:

  • By constraining your options, it helps you think about your data manipulation challenges.

  • It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate your thoughts into code.

  • It uses efficient backends, so you spend less time waiting for the computer.

The aim here isn’t to have the fastest dataframe library around (though it certainly helps that we’re building on Polars, one of the fastest).
Instead, we’re aiming to bridge the best of many worlds:

  • the elegance of dplyr
  • the speed of polars
  • the joy of Elixir

That means you can expect the guiding principles to be ‘Elixir-ish’. For example, you won’t see the underlying data mutated, even if that’s the most efficient implementation. Explorer functions will always return a new dataframe or series.

Links:

https://github.com/elixir-explorer/explorer

Acknowledgements

Explorer is an extensive library and there’s much more we could say. But for now, we’d just like to thank the dozens of contributors who’ve added wonderful improvements over the years. :heart:

16 comments
#library#explorer
1818 16

Most Liked

billylanchantin

billylanchantin

Explorer - version 0.8

Explorer has released version 0.8!

Added

  • Add explode/2 to Explorer.DataFrame. This function is useful to expand the contents of a {:list, inner_dtype} series into a “inner_dtype” series.

  • Add the new series functions all?/1 and any?/1, to work with boolean series.

  • Add support for the “struct” dtype. This new dtype represents the struct dtype from Polars/Arrow.

  • Add map/2 and map_with/2 to the Explorer.Series module.
    This change enables the usage of the Explore.Query features in a series.

  • Add sort_by/2 and sort_with/2 to the Explorer.Series module.
    This change enables the usage of the lazy computations and the Explorer.Query module.

  • Add unnest/2 to Explorer.DataFrame. It works by taking the fields of a “struct” - the new dtype - and transform them into columns.

  • Add pairwise correlation - Explorer.DataFrame.correlation/2 - to calculate the correlation between numeric columns inside a data frame.

  • Add pairwise covariance - Explorer.DataFrame.covariance/2 - to calculate the covariance between numeric columns inside a data frame.

  • Add support for more integer dtypes. This change introduces new signed and unsigned integer dtypes:

    • {:s, 8}, {:s, 16}, {:s, 32}
    • {:u, 8}, {:u, 16}, {:u, 32}, {:u, 64}.

    The existing :integer dtype is now represented as {:s, 64}, and it’s still the default dtype for integers. But series and data frames can now work with the new dtypes. Short names for these new dtypes can be used in functions like Explorer.Series.from_list/2. For example, {:u, 32} can be represented with the atom :u32.

    This may bring more interoperability with Nx, and with Arrow related things, like ADBC and Parquet.

  • Add ewm_standard_deviation/2 and ewm_variance/2 to Explorer.Series.
    They calculate the “exponentially weighted moving” variance and standard deviation.

  • Add support for :skip_rows_after_header option for the CSV reader functions.

  • Support {:list, numeric_dtype} for Explorer.Series.frequencies/1.

  • Support pins in cond, inside the context of Explorer.Query.

  • Introduce the :null dtype. This is a special dtype from Polars and Apache Arrow to represent “all null” series.

  • Add Explorer.DataFrame.transpose/2 to transpose a data frame.

Changed

  • Rename the functions related to sorting/arranging of the Explorer.DataFrame.
    Now arrange_with is named sort_with, and arrange is sort_by.

    The sort_by/3 is a macro and it is going to work using the Explorer.Query module. On the other side, the sort_with/2 uses a callback function.

  • Remove unnecessary casts to {:s, 64} now that we support more integer dtypes.
    It affects some functions, like the following in the Explorer.Series module:

    • argsort
    • count
    • rank
    • day_of_week, day_of_year, week_of_year, month, year, hour, minute, second
    • abs
    • clip
    • lengths
    • slice
    • n_distinct
    • frequencies

    And also some functions from the Explorer.DataFrame module:

    • mutate - mostly because of series changes
    • summarise - mostly because of series changes
    • slice

Fixed

  • Fix inspection of series and data frames between nodes.

  • Fix cast of :string series to {:datetime, any()}

  • Fix mismatched types in Explorer.Series.pow/2, making it more consistent.

  • Normalize sorting options.

  • Fix functions with dtype mismatching the result from Polars.
    This fix is affecting the following functions:

    • quantile/2 in the context of a lazy series
    • mode/1 inside a summarisation
    • strftime/2 in the context of a lazy series
    • mutate_with/2 when creating a column from a NaiveDateTime or Explorer.Duration.

Contributors

Thank you to everyone who opend up a PR:

And thank you to the first-time contributors!:

Changelogs

Full Changelog: Comparing v0.7.2...v0.8.0 · elixir-explorer/explorer · GitHub
Official Changelog: Changelog — Explorer v0.11.1

10
Post #2
billylanchantin

billylanchantin

[Release] Explorer - v0.10

Added

  • Add support for the decimals data type.

    Decimals dtypes are represented by the {:decimal, precision, scale} tuple,
    where precision can be a positive integer from 0 to 38, and is the maximum number
    of digits that can be represented by the decimal. The scale is the number of
    digits after the decimal point.

    With this addition, we also added the :decimal package as a new dependency.
    The Explorer.Series.from_list/2 function accepts decimal numbers from that
    package as values * %Decimal{}.

    This version has a small number of operations, but is a good foundation.

  • Allow the usage of queries and lazy series outside callbacks and macros.
    This is an improvement to functions that were originally designed to accept callbacks.
    With this change you can now reuse lazy series across different “queries”.
    See the Explorer.Query docs for details.

    The affected functions are:

    • Explorer.DataFrame.filter_with/2
    • Explorer.DataFrame.mutate_with/2
    • Explorer.DataFrame.sort_with/2
    • Explorer.DataFrame.summarise_with/2

  • Allow accessing the dataframe inside query.

  • Add “lazy read” support for Parquet and NDJSON from HTTP(s).

  • Expose more options for Explorer.Series.cut/3 and Explorer.Series.qcut/3.
    These options were available in Polars, but not in our APIs.

Fixed

  • Fix creation of series where a nil value inside a list * for a {:list, any()} dtype -
    could result in an incompatible dtype. This fix will prevent panics for list of lists with
    nil entries.

  • Fix Explorer.DataFrame.dump_ndjson/2 when date time is in use.

  • Fix Explorer.Series.product/1 for lazy series.

  • Accept %FSS.HTTP.Entry{} structs in functions like Explorer.DataFrame.from_parquet/2.

  • Fix encode of binaries to terms from series of the {:struct, any()} dtype.
    In case the inner fields of the struct had any binary (:binary dtype), it was
    causing a panic.

Changed

  • Change the defaults of the functions Explorer.Series.cut/3 and Explorer.Series.qcut/3
    to not have “break points” column in the resultant dataframe.
    So the :include_breaks is now false by default.

Contributors

New Contributors

Changelogs

8
Post #5
billylanchantin

billylanchantin

[Blog] Explorer 0.8: The dtype release

https://cigrainger.com/explorer-0-8-0/

5
Post #3

Last Post!

tmbb

tmbb

Ok, fair enough, no complex numbers going back and forth. Which makes sense, given how rarely complex numbers come up in real datasets of the kind Explorer is used to.

I understand, but do you really plan o having different backends for Explorer? I don’t think that’s a likelye possibility and I was assuming you’d want to keep using Polars, despite ir being decoupled in practice.

It would only be “trivial” if I can rely on the fact that a dataframe consists of a number of ChunkedArrays, which again is specific to the Polars backend. The only way to convert data form Explorer to SciEx “cheaply” is by assuming something about the layout of the data on the rust side.

And in any case, one of the problems is that becuase Polars and SciEx are compiled independently, they may actually use different rust compilers or simply different memory representations. Any kind of data conversion would have to serialize the data to a file (as you suggest) or use a fixed representation which could be read from the other side.

I guess I like the serialization idea and I’ll pursue it.

0
Post #17

Where Next?

View thread on forum (has 16 responses!)
library
explorer
Home News>Announcing
#library#explorer
59 1818 16
Last post

Popular in Announcing Top

Qqwy
News>Announcing

MapDiff: A small library that computes the difference between two maps
Hello everyone, I wrote a small library today called MapDiff. It returns a map listing the (smallest amount of) changes to get from map...
#library#maps#mapdiff
36 8904 15
New
jakub-zawislak
News>Announcing

Formex - a form library for Phoenix inspired by Symfony
Hi everyone, I’m coming from the Symfony (PHP) framework. I like Phoenix, but it has a one thing that was build much better in the Symfo...
/phoenix#forms#library#formex
92 13137 43
New
pkrawat1
News>Announcing

Gringotts: A complete payment library for Elixir and Phoenix Framework
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
#library#gringotts#payments#payment-gateways
193 12944 51
New
danschultzer
News>Announcing

Pow - Robust, modular, extendable user authentication and management system
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
/phoenix#authentication#library#authorization#user_system#pow
548 30338 241
New
aesmail
News>Announcing

Kaffy - a quick and flexible admin interface for phoenix applications
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
/phoenix#ecto#library#admin-tools#kaffy
301 24693 97
New
kevinlang
News>Announcing

Ecto_sqlite3 - an Ecto3 SQLite3 adapter
Hey all, We have made an Ecto3 Adapter for SQLite3, ecto_sqlite3! We have successfully on-boarded the full suite of integration tests (...
#ecto#library#ecto_sql#ecto_sqlite3
153 8138 52
New
Flo0807
News>Announcing

Backpex - a highly customizable admin panel for Phoenix LiveView applications
Hello everyone! I am excited to share our heart project Backpex with you. After building several Phoenix applications, we realized that...
/phoenix#library#liveview#admin-tools#phoenix-live-view#phoenix_live_view#live-view#phoenix-liveview#backpex
291 10879 90
New

Other popular topics Top

lanycrost
Questions & Help>Questions

How To Implement if...else if...else condition
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
#how-to-question
40 52473 6
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 29159 36
New
joeerl
Questions & Help>Questions

Upgrading Elixir - how to check versions, delete, and upgrade?
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
#upgrade
63 41802 13
New
AstonJ
Chat & Discussions>Discussions

LiveView demos, examples, and sample apps thread!
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
/phoenix#liveview
232 30935 60
New
JorisKok
Questions & Help>Questions

How is it possible to get 2 million websocket connections when you have 65536 available ports?
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
/phoenix
20 19385 4
New
jason.o
Questions & Help>Questions

No such input `xxxxx` for action ResourceName1.create
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
/ash
3 78754 2
New
Back to top

News>Announcing

News Announcing ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Honeybadger

Move fast and fix things! Error tracking with less noise for Elixir devs who love to ship. Start your free dev account.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement