This is an interesting thread. Would be exciting to further explore Machine Learning/Deep Learning and Big Data with Elixir.
I have heard from Francesco Cesarini in a video in youtube that number crunching may not be a strong area of Erlang. @ssagaert 's post is interesting on that account. I guess acknowledging that aspect or perhaps doing something or advocating improvements in that area for Erlang/Elixir would be something if Erlang/Elixir would be used way more in Distributed Data Computing
I wonder what avenues have been explored to improve Erlang VM’s performance for number crunching? I read that it may not have been a priority because of the lack of number crunching in telecom. Does that mean there is room for improvement, or is there an architectural problem that makes implementing fast matrices inexorable?
While it would be nice to do everything in Elixir, for now we will have to offload the CPU/memory-intensive calculations to a specialized language such as Julia, Python, R, Mathematica, C, Rust, et al. I recently bumped into the Futhark language, which is “a statically typed, data-parallel, and purely functional array language, and comes with a heavily optimising ahead-of-time compiler that generates GPU code via OpenCL.” I like the idea of staying in a functional language, and of targeting the GPU even more!
As programmers, we should aim to use a higher and higher-level languages because it’s easier to write correct code (Rich Hickey said this was one of his reasons for creating Clojure). The industry average defect rate is “about 15 - 50 errors per 1000 lines of delivered code,” so we want to write fewer lines of code for more reliable programs. The suckless community takes this idea to the extreme.
“One of my most productive days was throwing away 1,000 lines of code.” — Ken Thompson
We are wondering if we can we write similar Big Data tools in fewer lines of code using Elixir. And it seems that way because OTP has mostly solved the big and distributed part of the problem.
One point I liked in that blog post was that larger codebases tend to move toward service-oriented or microservice architectures. This makes sense to me as a way of isolating defects and reducing leaky interfaces. So why bet everything on Spark, which has MLlib, Streaming, YARN, GraphX, etc, all in one codebase? It is an impressive tool, and certainly convenient, but that’s not enough reason to not try to do it better, differently.
I wholeheartedly agree!
I have read somewhere just a few weeks ago that most libraries used for data science (extreme number crunching) use bindings to basically the same Fortran code. I’m struggling to find the quote, but it was in some discussion about Julia or Rust for data science vs R, Python, Matlab.
I wonder what would be the story for a numpy in Erlang/Elixir with nifs or ports, for instance. There’s the whole deal with reductions and the possibility that delegating too much work to number crunching code in other languages hurts reduction count and Erlang’s scheduler, which is IMO the real reason why Phoenix beat everyone else in the Phoenix Showdown benchmarks. It’s something I’d like to experiment with, but don’t have the time right now.
There is NumEr, which is a collection of Erlang NIF functions for BLAS operations on vectors and matrices with CUDA, but CUDA has an overhead and must be used wisely.
I’ve been playing with rustler, it allows you to create NIFs in Rust. My intention is to create a “NumEx” to use real matrices from Elixir. While I am not 100% sure of Rust because of it also being a new language, but I find it as a language much pleasing than C/C++. Rustler is also the easiest way I’ve found to create a NIF.
Hello,
I started playing around with building a library that mimics Apache Storm, but in Elixir and much simpler. For those unfamiliar, Apache Storm lets you build complex job topologies/pipelines and runs them distributedly.
Currently, it’s capable of doing things like distributed word count and distributed join. When I say “distributed”, I really mean parallel (all processes run on a single BEAM VM for now).
The README gives a pretty good explanation of what it does and how it works:
https://github.com/cjbottaro/tempest
Also see the examples:
So I just watched the ElixirConf.EU keynote and I’m a bit unclear on the uses of GenBroker/GenStage. Are those things that would supercede what I’m building, or are they things that would make what I’m building easier?
The keynote depicts pipelines as linear/straight, where as my lib lets you build a directed, acyclic graph, rather than just a straight pipeline. Is my understanding of GenBroker/GenStage correct, or would they allow you make complex graphs of pipelines?
Also, is there any interest in a library/framework like this? The motivation is that Storm, like Spark, like Hadoop are difficult to install/use and are generally extremely heavy with lots of operational dependencies, etc. I want something extremely simple use, albeit at the expense of crazy fault tolerance requirements and maybe performance. You know… “Medium Data” problems… 
What a cool project (and cool name)! and exactly what I was thinking about when I posted this topic. The examples are so easy to comprehend because of Elixir’s expressiveness and the clean interfaces.
I’m sure this library would be useful to Alchemists with “Medium Data” problems I would encourage you to release a Hex package for it. Fault-tolerance and performance can come later as more people depend on the library.
I haven’t watched the keynote about GenBroker/GenStage but I too am curious about their applications in this sense.
We all like Elixir but using it for “Big data” would be a stretch BEAM is not optimized (and realisticly can not be optimized) for such workloads.
There is Apache Storm, Apache Spark, Apache Flink, Apache Apex and there is also something called Apache Beam → some top layer over existing platforms so you can easy move flow between them. Summary so quite a lot to play with
There will be course Big Data Analysis with Scala and Spark - free
On the BEAM not doing jit on the machine code… I thought I saw a roadmap for Erlang from a core guy saying they were working on jit using llvm for a future release. Possibly it was Erlang Factory SF? (on my phone so not looking too hard)