dyowee23

dyowee23

Using Elixir for Data Science and Machine Learning

Will future versions of Elixir be appropriate for data science/machine learning projects, like Scala or F#?

Most Liked

Hoegbo

Hoegbo

We are using Elixir AI systems in production. Elixir/Erlang is fantastic for it. We could use more native libraries there are a 4-5 that I know of. and there is a guy that has 3 libraries on hex. we rolled our own stuff though Hoping to be able to open source the backbone of that eventually.

Only downside is the lack of libraries compared to certain other languages. But since the BEAM is awesomeness incarnated making your own stuff is fairly easy. Break everything into small enough pieces and performance issues with regards to calculations will not be hindering you at all. I could write wall to wall about how Awesome Elixir/Erlang is for AI and especially neural networks.

27
Post #6
cgarciae

cgarciae

Here is my experience so far:

  1. Two month ago I had to process a big CSV file, transform it, and write the output to another file. Python with Pandas was running out of memory because of some operations that required pure python code, so I gave Elixir a shot. The good thing is that the system didn’t freeze due lack of memory, the whole process a single parallel pipeline (as most as it could), and it was very easy to program. The bad thing is that it was SLOWWWWWW. Switched to Spark, Scala + RDDs made the problem look easy.

  2. This month I’ve been playing around with TensorFlow. TensorFlow runs in C++ so it helps you with performance even if you are on Python, I also take it uses all the cores of CPU to that helps. But last week I implemented a CNN for MNIST and started to feel the weight of the model, so now I am now looking to run the computation in the GPU to gain some performance.

My thoughts on the matter: Elixir would be an awesome language for Machine Learning and Big Data, but it still lacks: 1) Libraries with C / C++ / Rust? bindings through NIFs or Ports to perform high performance computation, we need need the equivalent of Numpy, Pandas, and TensorFlow, 2) the attention of the scientific community, who’s currently in Python and some are migrating to Julia.

I’d be interested to hear concrete examples from @Hoegbo because the ErlangVM is not good at number crunching, you can have each layer as a process if you like but that isn’t going to get you anywhere if you don’t have BLAS or CUDA. Hopefully one day it does.

14
Post #7

Last Post!

marciol

marciol

Hei @ericsteen, what is the landscape now that we have Nx at our disposition?

Where Next?

Popular in Discussions Top

ben-pr-p
In general I’ve been sticking to this community style guide GitHub - christopheradams/elixir_style_guide: A community driven style guide ...
New
und0ck3d
Hello everyone! A few days ago I’ve created a topic here about how people were creating CMSs with Elixir and Phoenix. I’ve been studying...
New
pdgonzalez872
If this has been asked here before, please point me to where it was asked as I didn’t find it when I searched the forum. Maybe a mailing ...
New
mikl
I wanted to capitalize a string, and tried using String.capitalize(). That generally works well, until you try to capitalize a word like...
New
opsb
We’re considering our architecture from a viewpoint of scaling our traffic heavily over the next 6 months. Our current deployment is runn...
New
marciol
Please, let me know if this kind of discussion already took place in another topic . Hi all, how do you consider if is better to build ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New

We're in Beta

About us Mission Statement