dyowee23
Using Elixir for Data Science and Machine Learning
Will future versions of Elixir be appropriate for data science/machine learning projects, like Scala or F#?
Most Liked
Hoegbo
We are using Elixir AI systems in production. Elixir/Erlang is fantastic for it. We could use more native libraries there are a 4-5 that I know of. and there is a guy that has 3 libraries on hex. we rolled our own stuff though Hoping to be able to open source the backbone of that eventually.
Only downside is the lack of libraries compared to certain other languages. But since the BEAM is awesomeness incarnated making your own stuff is fairly easy. Break everything into small enough pieces and performance issues with regards to calculations will not be hindering you at all. I could write wall to wall about how Awesome Elixir/Erlang is for AI and especially neural networks.
swennemans
cgarciae
Here is my experience so far:
-
Two month ago I had to process a big CSV file, transform it, and write the output to another file. Python with Pandas was running out of memory because of some operations that required pure python code, so I gave Elixir a shot. The good thing is that the system didn’t freeze due lack of memory, the whole process a single parallel pipeline (as most as it could), and it was very easy to program. The bad thing is that it was SLOWWWWWW. Switched to Spark, Scala + RDDs made the problem look easy.
-
This month I’ve been playing around with TensorFlow. TensorFlow runs in C++ so it helps you with performance even if you are on Python, I also take it uses all the cores of CPU to that helps. But last week I implemented a CNN for MNIST and started to feel the weight of the model, so now I am now looking to run the computation in the GPU to gain some performance.
My thoughts on the matter: Elixir would be an awesome language for Machine Learning and Big Data, but it still lacks: 1) Libraries with C / C++ / Rust? bindings through NIFs or Ports to perform high performance computation, we need need the equivalent of Numpy, Pandas, and TensorFlow, 2) the attention of the scientific community, who’s currently in Python and some are migrating to Julia.
I’d be interested to hear concrete examples from @Hoegbo because the ErlangVM is not good at number crunching, you can have each layer as a process if you like but that isn’t going to get you anywhere if you don’t have BLAS or CUDA. Hopefully one day it does.
Last Post!
marciol
Popular in Discussions
Other popular topics
Chat & Discussions>Discussions
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security










