Hi @ryan-duve, welcome!

You are right that the concurrency aspect of Elixir won’t help much with data exploration. In this area, we are pretty much on the same baseline as Python. The good news is that we can also delegate to C/Rust, just as Python, and that’s the goal with Explorer. It is based on the amazing Polars (which also has Python bindings) and folks report it to be consistently faster than Pandas. So my suggestion is to give Explorer a try and let us know what you are missing in the issues tracker. It is pretty much work in progress and you will hit some walls. But this feedback will help us know what to focus on next. Also, the next release of Livebook will start bringing more features guided towards data exploration.

For machine learning inference, one of the benefits of Elixir is that you can run it within your application. I cover this in the Bumblebee announcement. If this is useful or not, it will be depend on your application. For example, for data processing with Broadway or embedded devices with Nerves, that can be really useful. But for a web application, you may resort to separate services anyway, especially for larger models, just as you would in Python.

In other words, for your questions the answers are mostly “just do it as you have been doing in Python” - unless you see a benefit on deploying the model with your app. So you may be wondering where Elixir can be beneficial:

• Using a functional programming language for writing computational models can have its benefits: Nx (Numerical Elixir) is now publicly available - Dashbit Blog

• Nx was designed from scratch to support multiple compilers. This means that you can choose between Torch or XLA easily. Most of our focus is on XLA at the moment but this vision will become truer as we pour more work into it

• By using Nx as a foundation, everything we build can be compiled to the GPU. For example, we are working on Scholar, which is equivalent to scikit-learn, but everything can be tensor compiled to both CPU and GPU and we get consistently faster results thanks to it

• Finally, when it comes to distributed and federated work, then Elixir has the potential to standout. We want to explore how to coordinate work across multiple machines for both data processing and machine learning

I hope this gives you an idea of where we are and what to expect. There is a lot of work but there is a group of people who are working hard and are excited on making this vision come true. Questions, feedback, and help are all welcome!