With blockchain technology on the rise and bigdata being where it already is, numerical crunching using native libraries should be a part of Elixir’s toolkit. I want a toolkit so that what I have in mind for the long term becomes feasible without having to invest in other languages.
@josevalim - I am sure you have an architecture in mind to execute this, or you have robust reasons as to why this is the “wrong thing” for Elixir. Either way I would really appreciate your inputs. It looks like the individual parts are available but not the sum of things. Is Numerical Crunching in Elixir missing some effort (development + architecture + supervisory) to fit things together ? Or is it that there are certain aspects of Erlang VM that make this a fool’s errand ?
First list is list of capabilities that exist - straight forward
Second list is list of capabilities that do not exist - From what I read on the forums, Elixir / Erlang lacks the following capabilities in order to be a contender in numerical processing.
Capability Exists:
INPUT/OUTPUT data from disparate non DB sources
- CSV @ https://github.com/meh/cesso
- Excel reader @ https://hex.pm/packages/xlsxir
- Excel writer @ https://hex.pm/packages/elixlsx
Number crunching - arrays @ https://github.com/Qqwy/elixir-arrays - stable ? fast ?
- tensors @ https://github.com/Qqwy/tensor - stable ? fast ?
- Pandas-style Dataframe @ https://github.com/jordipolo/dataframe - abandoned ?
- NumEr @ https://github.com/vascokk/NumEr // vector and matrix operations with CUDA - abandoned ?
Spark-style Distributed processing - Does Elixir already possess this ?
- Does Elixir need an API for Spark ?
- Spark @ https://github.com/cjbottaro/tempest - abandoned ?
- LFE (Lisp Flavoured Erlang Wrapper Library for SciPy, NumPy, etc) @ https://github.com/lfex/lsci - exists for LFE, not Elixir. Is it useful ?
Capability does not exist:
Native NumPy/SciPy
- Possible via Python via ErlPort ?
- Possible via CUDA ? @ Big Data with Elixir
- Possible via BLAS ?
- Matlang for Erlang ?
- Generic Research for natively extending Elixir / Erlang using Rust
** Write Elixir extensions in Rust ? WITH
** Rust support ndarry similar to numpy @ https://github.com/alexchandel/array-rs
** Rust API for numpy @ https://github.com/termoshtt/rust-numpy WITH
** Safe Rust bridge for creating Erlang NIF functions - https://github.com/hansihe/Rustler
** Rewrite Pandas code in Rust ?
Erlang VM JIT support ? @ http://www.erlang-factory.com/sfbay2016/kenneth-lundin - ETA ?
Other references from this forum:
- Number crunching ? @ Using Elixir for Data Science and Machine Learning
