dyowee23

dyowee23

Using Elixir for Data Science and Machine Learning

Will future versions of Elixir be appropriate for data science/machine learning projects, like Scala or F#?

Most Liked

Hoegbo

Hoegbo

We are using Elixir AI systems in production. Elixir/Erlang is fantastic for it. We could use more native libraries there are a 4-5 that I know of. and there is a guy that has 3 libraries on hex. we rolled our own stuff though Hoping to be able to open source the backbone of that eventually.

Only downside is the lack of libraries compared to certain other languages. But since the BEAM is awesomeness incarnated making your own stuff is fairly easy. Break everything into small enough pieces and performance issues with regards to calculations will not be hindering you at all. I could write wall to wall about how Awesome Elixir/Erlang is for AI and especially neural networks.

27
Post #6
cgarciae

cgarciae

Here is my experience so far:

  1. Two month ago I had to process a big CSV file, transform it, and write the output to another file. Python with Pandas was running out of memory because of some operations that required pure python code, so I gave Elixir a shot. The good thing is that the system didn’t freeze due lack of memory, the whole process a single parallel pipeline (as most as it could), and it was very easy to program. The bad thing is that it was SLOWWWWWW. Switched to Spark, Scala + RDDs made the problem look easy.

  2. This month I’ve been playing around with TensorFlow. TensorFlow runs in C++ so it helps you with performance even if you are on Python, I also take it uses all the cores of CPU to that helps. But last week I implemented a CNN for MNIST and started to feel the weight of the model, so now I am now looking to run the computation in the GPU to gain some performance.

My thoughts on the matter: Elixir would be an awesome language for Machine Learning and Big Data, but it still lacks: 1) Libraries with C / C++ / Rust? bindings through NIFs or Ports to perform high performance computation, we need need the equivalent of Numpy, Pandas, and TensorFlow, 2) the attention of the scientific community, who’s currently in Python and some are migrating to Julia.

I’d be interested to hear concrete examples from @Hoegbo because the ErlangVM is not good at number crunching, you can have each layer as a process if you like but that isn’t going to get you anywhere if you don’t have BLAS or CUDA. Hopefully one day it does.

14
Post #7

Last Post!

marciol

marciol

Hei @ericsteen, what is the landscape now that we have Nx at our disposition?

Where Next?

Popular in Discussions Top

AstonJ
I’ve just started the Phoenix part of the utterly brilliant online course by @pragdave. On generating the Phoenix app he uses the --no-ec...
New
und0ck3d
Hello everyone! A few days ago I’ve created a topic here about how people were creating CMSs with Elixir and Phoenix. I’ve been studying...
New
jer
I’ve been using umbrellas for a while, and generally started off (on greenfield projects at least) by isolating subapps based on clearly ...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
owaisqayum
I have a sample string sentence = "Hello, world ... 123 *** ^%&*())^% %%:>" From this string, I want to only keep the integers, ...
New
mmmrrr
Just saw that dhh announced https://hotwire.dev/ Is it just me or is this essentially live view? :smiley: Although I like the “iFrame-e...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Other popular topics Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40082 209
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42576 114
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

We're in Beta

About us Mission Statement