We can actually start collecting basic design conditions (tools, target size of the data, deploying hypothesis, etc) and the links from our different posts into a repository. Are you ok if I start it on Github with a README? We can write there a kind of “state-of-the-art” as introduction to see what can be done or planned.
tuned
ejc123
AstonJ
Don’t forget you can make Wiki’s here too (which anyone at Trust Level 1 or higher can edit). If @uranther is ok with it we can make the first post here a Wiki and you could all maintain it.
The main benefits are that it may encourage more discussion, shows up if anyone searches for it and it’s easier to make changes to (no need for PRs needing to be approved, etc). Up to you 
4
mkunikow
I don’t think so you can beat something like Apache Spark (Spark SQL, Spark Streaming, Machine Learning, GraphX).
But what would be nice some kind of interoperability for example elixir → Apache Spark or Apache Spark → Elixir.
2
uranther
They have many more man-years of development invested, too. Because of OTP, the same functionality can be duplicated in much less Erlang/Elixir code and have better scalability and reliability.
I agree! In the meantime of some major Elixir Big Data projects, interoperability would be great.
5
ssagaert
I think you are sadly mistaken: something like Spark cannot be duplicated in Elixir, easily or painfully.
The reason : Elixir ( & Erlang) are simply not suited for scientific programming/data science programming because they do not have mutable arrays and the corresponding linear algebra & machine learning & DSP libraries that build on top of that. Without mutable (large & possibly distributed) arrays it’s simply not possible to do any efficient number crunching.
Big Data is not just about accessing lot’s of data (files on HDFS or in various NoSQL dbs → this can be done in Elixir) but it really is what you do with that data once you’ve accessed it and that’s where data science/machine learning comes in.
There was one promising project called Matlang for Erlang by one ph.d. guy but that seems dead. It was based on wrapping the native BLAS matrix library just like matlab, sciPy, etc do.It showed decent performance. One of the reasons this never became mainstream is that it required a patch to the Erlang VM for efficiciently calling BLAS and that patch was rejected by the Erlang team.
Also the Erlang VM doesn’t do any jitting to native code like the JVM does, so (scalar) numerical performance & tight loops to iterate over collections of numbers is not great (without wrapping a native numerical lib).
7
tuned
@ssagaert
Very interesting post.
We are talking in fact to create a possible synergy between Elixir and Python in the same pipe. Leveraging Elixir to access HDFS-like and inter-operate to Python for the map-reduce or machine-learning, and back again to Elixir for the database and Web inter-operability. This can be done by using protobuffer or a common swap space.
Again, one of the weakness of the software you mentioned is of being monolithic and to enforce the use of certain tools (above all Java).
Python can have some overhead problems at performing as the actual tools on the market, so creating a pipeline with the characteristics above can be a great plus for Python users (that are the great majority of data scientists) not using the Apache pipe.
I didn’t say, and the other the same, it’s easy or a few weeks work, but there is quite a demand to “ease” the actual big-data pipe.
2
ssagaert
I don’ t think wrapping Python with Elixir is a very good Idea. Python is slow and already a glue language so that’s like 2 glue layers. For performance you should interface with a faster lower level language.
“there is quite a demand to “ease” the actual big-data pipe.”
I’m not sure this is true. Have you even studied Spark? You can program it already in 4 languages: Scala (the native language it’s programmed in. This is actually a high level language but fast (& also complex unfortunately)), Java, R & Python. So for the Pythonistas: there really is no need to recreate Spark in Python since it already has a Python interface. Next to that you do do a lot of data retrieval/manipulation in SQL (for non-prgrammers) & there are interactive notebooks (like in iPython).
Nothing prevents you in principle to develop an Elixir interface to Spark but to make this workable you would need efficient (mutable) datatstructures to represent dataframes & large arrays (vectors/matrices/tensors) in Elixir and this does not exist at the time. And no a (nested) (linked) list is NOT going to cut it for representing vectors/matrices/tensors in Elixir.
6
tuned
I didn’t say anything about wrapping… but creating a data pipe that involves both.. with common data-exchange format. Each one doing what it can do at best. I have used many times the word ‘inter-operability’ but never ‘wrapping’ I think.
What Spark does is a part of what a pipeline does. It’s a part in the pipeline. We are talking about the possibility of using Python and Elixir to build an ALTERNATIVE pipeline. Please see the posts above.
Just a note, a market with one alternative is not really choosing a tool, it’s undergoing a tool. Finally, the approach of saying “there is already another tool” doesn’t work in Open Source because every tool is shaped to solve a problem and never to solve every problem; there is always space for a tool doing the same thing but in a different way.
2