Data science and machine learning workflows in Elixir

Hi @ryan-duve, welcome!

You are right that the concurrency aspect of Elixir won’t help much with data exploration. In this area, we are pretty much on the same baseline as Python. The good news is that we can also delegate to C/Rust, just as Python, and that’s the goal with Explorer. It is based on the amazing Polars (which also has Python bindings) and folks report it to be consistently faster than Pandas. So my suggestion is to give Explorer a try and let us know what you are missing in the issues tracker. It is pretty much work in progress and you will hit some walls. But this feedback will help us know what to focus on next. Also, the next release of Livebook will start bringing more features guided towards data exploration.

For machine learning inference, one of the benefits of Elixir is that you can run it within your application. I cover this in the Bumblebee announcement. If this is useful or not, it will be depend on your application. For example, for data processing with Broadway or embedded devices with Nerves, that can be really useful. But for a web application, you may resort to separate services anyway, especially for larger models, just as you would in Python.

In other words, for your questions the answers are mostly “just do it as you have been doing in Python” - unless you see a benefit on deploying the model with your app. So you may be wondering where Elixir can be beneficial:

• Using a functional programming language for writing computational models can have its benefits: Nx (Numerical Elixir) is now publicly available - Dashbit Blog

• Nx was designed from scratch to support multiple compilers. This means that you can choose between Torch or XLA easily. Most of our focus is on XLA at the moment but this vision will become truer as we pour more work into it

• By using Nx as a foundation, everything we build can be compiled to the GPU. For example, we are working on Scholar, which is equivalent to scikit-learn, but everything can be tensor compiled to both CPU and GPU and we get consistently faster results thanks to it

• Finally, when it comes to distributed and federated work, then Elixir has the potential to standout. We want to explore how to coordinate work across multiple machines for both data processing and machine learning

I hope this gives you an idea of where we are and what to expect. There is a lot of work but there is a group of people who are working hard and are excited on making this vision come true. Questions, feedback, and help are all welcome!

I could imagine that a lot of the data exploration is ad-hoc, and that’s where Livebook really helps. It gives you a nice playground with the added bonus of documenting your process and sharing back and forth with your developer colleagues.

I’ve also had nice results with Flow, which allows you to define a concurrent data-crunching pipeline, within Livebook. No better feeling than setting up a pipeline that utilises all your resources just right, and being able to tinker with it in a short feedback loop.

Add to this the ability to use Ecto within Livebook, and you’re all set up to load and dump to the database at full speed.

And then, at the end of the ride, you can graph everything nicely with the kino-vegalite package to impress everyone with your colourful results.

I can’t add anything to the ML side of things, except the interop between Livebook and all the ML libraries mentioned is already great, and getting better. Welcome to the community!

Hi,
Possibly late, sorry to dig up this post but in case you haven’t come across this book yet?
https://pragprog.com/titles/smelixir/machine-learning-in-elixir/
Best regards