Machine Learning & Elixir

I feel this old (2009) stackoverflow thread on numerical computing in Erlang is relevant to the topic. I also think that the main feature of a language to be suitable for ML is vectorization, is Elixir good for vectorization of computations?

Interesting talk, it seems however that BEAMJIT has been on its way for a long time.

Garret Smith has been working on a system that wraps the Tensor Flow python libraries and implements an Erlang system around monitoring the processes. Check out http://guild.ai

Is it possible for elixir to just be the controller for python processes? Since python has all the available libraries for easily working with data and it crunches numbers faster than elixir could python scripts do the work and pass back variables to an elixir app that is an API?

Easily yep, or go even lower level and access the base C/C++ API as well via a port or so. But yep, Python interacts wonderfully via a port, I’ve used it many times.

You mean you’ve accessed python via REST call from elixir? ex: API in Django

Seems more like Ports and Port Drivers — Erlang System Documentation v29.0.2.

You might want to check out The Erlangelist - Outside Elixir, https://hackernoon.com/mixing-python-with-elixir-7a2b7ac6696, and http://erlport.org/ as well.

I can second @idi527 's erlport recommendation.

As for elixir+python for ml, we are calling python from Elixir for tensorflow usage, though we are also evaluating tensorflex for running the trained networks. Good times

Where I work, we apply Machine Learning/Deep Learning to various problems in the retail space. Leveraging Elixir and BEAM for ML/DL is something I’m also curious about. I’ve been looking at some of the available Elixir wrappers/libraries, such as Tensorflex.

Although Elixir and BEAM might not be ideal for doing the heavy number-crunching, I imagine that it could be a great technology for building a platform to facilitate for distributed training of neural networks. Some kind of worldwide machine learning orchestration platform/service.

Imagine a peer-to-peer-like platform where people could volunteer idle CPUs and GPUs. Members of the network could then use idle hardware on demand for training their neural nets. One could implement some kind of point-based system for fair time-sharing of hardware resources in the network, etc.

The general idea of distributed computing is nothing new, as it has been applied to many different fields (Folding@Home being an excellent example).

Something that might be interesting for interfacing python is Pyrlang. However, iirc it doesn’t work on 20 yet.