Using Elixir for Data Science and Machine Learning

Ports are extremely safe. They are, in principle, a mechanism that enables you to treat an OS process as an Erlang process. This means you can monitor them, link, etc just like an Erlang/Elixir process.

I know of at least one team using python to do ML that way - have python scripts doing ML and call them from Elixir using ports.

You might also find this talk from ElixirConf EU interesting - it talks about using Elixir for controlling Python, Julia and R from Elixir.

@michalmuskala so using ErlPorts would work fine or is there a better way? Or would it have to be scripts that you pass variables to and from. I ask because I am trying to do this exact same thing. I want to use elixir to pretty much be the master controlling async processes on a bus. So that python processes could block other python processes through the elixir message bus. Or at least something like this.

I would be using Phoenix, but with a React front end, Ecto with postgreSQL, with python doing all the ML dirty work. Since python already has Tensorflow and Scikit Learn, numpy, and pandas, it is at least better at mathematical computations than Elixir.

@marcusjwhelan A port is treated from the Elixir/Erlang side as a process. This means you can send messages to them, and also receive messages from them. There exist slightly higher-level wrappers like ErlPort, that expose an API to call any exposed functions directly from the other language.

Setting up a connection between Elixir and Python using ErlPort is very easy (I did it before to call into some NLTK-stuff in Python from Elixir), and probably the direction you’d want to take.

@Qqwy Thank you, I have been asking so many people what the best approach is. Either they don’t know you can do this or they don’t know a good way to do it. I have seen people say ErlPort but also that it doesn’t work well all the time. Something about data loss or something along the lines of variables not matching up.

But here is my last final question. Can I have a python process working on an elixir thread block another python process on another thread through elixir’s messaging system?

Doesn’t ErlPort start a python interpreter for every process? I don’t quite remember. If it’s true, it might be not the most efficient way.

@idi527 Rather, ErlPort starts one python interpreter, which is one process (and which can be interacted with like any other process). You can indeed run multiple instances of these side-by-side but usually I don’t think there is a need to do this.

@marcusjwhelan

Can I have a python process working on an elixir thread block another python process on another thread through elixir’s messaging system?

I think this is only possible if the to-be-blocked python Port listens for messages, i.e. using cooperative scheduling. In-VM processes are managed by the scheduler(s) and are therefore scheduled preemptively. This is not the same for a Port (which is a connected Operating-System-level process, meaning that the Operating System also handles how its CPU-time scheduled).

I do wonder where this question comes from, as it seems like a weird and complex thing that might be avoided by structuring your application differently.

@Qqwy
I was simply curious to the possibility of doing this. But I see now that yes this can just be avoided. You shouldn’t need to have python process blocking since if you are going to need another language that doesn’t support the same concurrency model, forcing it to would be illogical.

I just had this crazy idea to have python processes create models and that you could have nodes of python scripts that only worked on a single model and if you wanted to add classifiers and create a larger neural network that could be dynamic. Something like this,
tensorflow playground where each column’s python scripts would not start to run till all of the previous columns finished. But that means it is synchronous anyways. So I was wrong to think I needed the ability to run multiple concurrent processes that needed to block one another.

I’m a data scientist who loves Elixir.

There is an incredible amount of potential for Elixir in data science and machine learning, since the language provides excellent facilities for data transformation through pattern matching, piping, etc.

Also, the functional programming paradigm is a much more natural fit for data science than object-oriented and imperative programming, being conceptually closer to the problem space.

At my workplace, we primarily use Python (but also some R and Scala) for data science, which are the de facto standards in this domain. The reason for that, I think, is because of some extensive and useful libraries, such as pandas, scikit-learn, TensorFlow (by Google), Keras and many others.

Elixir has a lot of catching up to do in the library department with respect to Python.

One other aspect of Elixir (and the BEAM in general) which is of interest to data science, is the way that it allows for easy distribution and scaling, at least on the CPU-side. However, I’m not sure how to interface with GPUs via Elixir (e.g. via CUDA), which is essential for effective machine learning.

At the moment, I’m looking into using Elixir for hosting and exposing pre-trained machine learning models (trained using TensorFlow and Kreas) to consuming applications via APIs, etc. @anshuman23 has already created Tensorflex for that purpose, which looks very promising.

It is also sensible to use Elixir for the aspects of data science which have to do with gathering and preparing the data set you need to train your models (aka “tidy data”).

I see that this post is from 2016 and quite a lot has probably changed since then. I’m curious to hear your experiences with using Elixir/Erlang for AI and neural networks since then, if you’re still at it.

I stumbled upon these amazing blog post by @TheQuengineer :

http://www.automatingthefuture.com/blog/2017/2/20/deep-learning-building-and-training-a-multi-layered-neural-network-in-elixir

http://www.automatingthefuture.com/blog/2016/11/30/training-elixir-processes-to-learn-like-neurons

Perhaps they will be useful to others who follow this thread.