I agree with the forum.
In order to process big amounts of data, you need a way to store it, and a way to process the data. The storage is solved with HDFS. Note that it’s not the only option, you can store it with ceph, GFS, etc. In fact, HDFS started as a way to replicate Google’s GFS.
Now, regarding processing, I still don’t get why Hadoop and Spark are so popular. As many mentioned, Java is not the most suitable language. It’s OK to build a nice web service easily, but it’s not the best language for high intensive data mining. The same applies to Spark and Scala. So, why Scala? My guess is that Scala introduces functional programming with Java syntax on top of JVM.
I believe that erlang and elixir are far superior. So, you don’t even need Hadoop for serious data mining. You just need a storage engine (MongoDB, Riak, CouchDB or whatever). If you want to operate on files, you just need a distributed FS, and I believe that Ceph for example is much better than HDFS. Then, you implement the entire mining in Erlang or Elixir, which provides much more than what Hadoop of Spark gives you (fault tolerance distributed system with native map reduce). If you need any need to do any data intensive stuff, you can implement them on C or C++, provide a C interface and called those functions from within Erlang/Elixir.
This makes much more sense that installing few different frameworks which want to emulate what you can do in a few lines of code. Unfortunately, nobody seems to implement big data and data mining this way.
But still, if you would, you can design datastructures nicely aligned and even bitpacked as you can in C, but you don’t have syntax support.
