Using Elixir for Data Science and Machine Learning

There are a few threads here that might be of interest to you: Search results for 'machine learning' - Elixir Programming Language Forum

There is also Handbook of Neuroevolution Through Erlang

can someone tell me about his experience with this book?
(its kinda expensive, but looks really interesting to me)

Its worth the money

its been some time since I plowed through it but overall its a great read. and should get you up and running with the basics. I`d recommend knowing more than basics of Erlang when reading through it though.

If AI and elixir/Erlang is what you want to do. the this is a mandatory book to read through in my opinion.

We are using Elixir AI systems in production. Elixir/Erlang is fantastic for it. We could use more native libraries there are a 4-5 that I know of. and there is a guy that has 3 libraries on hex. we rolled our own stuff though Hoping to be able to open source the backbone of that eventually.

Only downside is the lack of libraries compared to certain other languages. But since the BEAM is awesomeness incarnated making your own stuff is fairly easy. Break everything into small enough pieces and performance issues with regards to calculations will not be hindering you at all. I could write wall to wall about how Awesome Elixir/Erlang is for AI and especially neural networks.

Here is my experience so far:

1. Two month ago I had to process a big CSV file, transform it, and write the output to another file. Python with Pandas was running out of memory because of some operations that required pure python code, so I gave Elixir a shot. The good thing is that the system didn’t freeze due lack of memory, the whole process a single parallel pipeline (as most as it could), and it was very easy to program. The bad thing is that it was SLOWWWWWW. Switched to Spark, Scala + RDDs made the problem look easy.

2. This month I’ve been playing around with TensorFlow. TensorFlow runs in C++ so it helps you with performance even if you are on Python, I also take it uses all the cores of CPU to that helps. But last week I implemented a CNN for MNIST and started to feel the weight of the model, so now I am now looking to run the computation in the GPU to gain some performance.

My thoughts on the matter: Elixir would be an awesome language for Machine Learning and Big Data, but it still lacks: 1) Libraries with C / C++ / Rust? bindings through NIFs or Ports to perform high performance computation, we need need the equivalent of Numpy, Pandas, and TensorFlow, 2) the attention of the scientific community, who’s currently in Python and some are migrating to Julia.

I’d be interested to hear concrete examples from @Hoegbo because the ErlangVM is not good at number crunching, you can have each layer as a process if you like but that isn’t going to get you anywhere if you don’t have BLAS or CUDA. Hopefully one day it does.

@Hoegbo @cgarciae has there been any advance in ML with Elixir? Is there a way to have elixir maybe pass variables to and from python methods so python could do the number crunching but also be controlled by elixir’s concurrency model?

I’m looking for the same, as mention in this amazing talk

We need more libraries on Elixir for ML, I’m pretty sure than ML in Elixir has a lot of potential, but make libraries for ML is not an easy work

Well that is what I am saying. Why can’t we leverage elixir to use something like ErlPort or Apache Thrift to use python to crunch the numbers and simply use elixir as the controller of python methods using elixir to have method concurrency with the message system. Can this not be done?

Can be done? Yes

Its the right way to do it? I dont think, thats because ports on Erlang have a high implact is not correctly used and can crash the BEAM, but just for operations its another history I don’t think it will crash the BEAM, I think with some research we can build a good library for Elixir ML