Tensorflex: Tensorflow bindings for Elixir

Thank you for creating this library, @anshuman23.

I recently changed my position at work from business developer to data scientist and I’m in the process of digging into a bunch of tools for data wrangling and machine learning.

Python is one of the most common programming languages for data science, due to extensive and robust libraries, such as pandas, NumPy, scikit-learn, TensorFlow, Keras and TensorForce. Those are the primary tools that we use at my workplace at the moment.

I’ll be looking into using Tensorflex as an alternative for deploying machine learning models.

One thing I’m curious about: When you created Tensorflex, did you envision for it to become to Elixir sort of what Keras is to Python? That is, not just for model-serving, but also for training artificial neural networks, etc.—A more generic abstraction layer on top of TensorFlow, CNTK and Theano.

Elixir with it’s highly concurrent capabilities, focus on data transformation (e.g. data pipelines) and strong support for meta-programming, lends itself very well to data science and machine learning. It makes me very excited to see that other people have similar thoughts.

There is an incredible amount of potential for Elixir within data science and machine learning.

Hello!
@mat_garland_1 and others who might be interested, here is the link for the PR which showcases how to use an RNN-LSTM model for sentiment analysis in Tensorflex. I am going to be working on documentation and a blog post for this as well as the Inception model over the next week, but the description for this PR is sufficiently detailed to get a general idea in the meanwhile:
https://github.com/anshuman23/tensorflex/pull/25

Thank you so much @anshuman23, I’ll certainly check it out! Great work!

You could also start a slave node. Execute your NIFs within the slave. With slaves you can also achieve fault tolerance. You’d also avoid the need for C code scaffolding. Assuming you got the NIF from a 3rd party you could avoid maintaining non-elixir code within your codebase.

Hi @mat_garland_1! Yes Tensorflex supports Inference for any graph created with Tensorflow. Since you mentioned LSTMs, I’m actually working on an example at the moment which will cover RNNs and LSTMs with Tensorflex. It should be up on the Github repository in a day or so. Will also try and write a blog post if it helps

I am really looking forward to this, thanks @anshuman23 for taking this on board.

It would be perfect if one were able to hand off data to an Elixir process → Tensor flow / flex ML pipeline and then use that trained data. I am particularly interested in the text classification / LSTM / Inference areas of ML for bots. It sounds like this is exactly what you are working on?

You create a useful product, thank you! I recently found a similar project, it’s called extensor. Maybe you should join forces.

Alternatively you could implement the binding as a C-Node.
There is a package called Nodex.

From Erlang’s point of view, the C node is treated like a normal Erlang node.

This way you still have fault tolerance.

That’s about right, Rust gives safety within rust, so it could make the interface safer, but it won’t do anything for the code running in another language, and if it is a simple wrapping anyway, not much point in using Rust then.

But most every library we’d be wrapping is C/C++. Here’s where the FUD kicks in. Rust is a new and another thing I’d have to learn. Also, from my limited knowledge of Rust, that while you can call C/C++ you give up safety by doing so.

So while you could write your own Rust lib for the fast parts and use Rustler to help with Elixir bindings, I’m not sure Rust helps in this case.

But I hear about Rust so it would be nice for someone to set the record straight.