Sorry to bump this older thread but I couldn’t find any recent threads on big data/data science/machine learning. I’d like to play around with Spark/Hadoop with website I’m making with Elixir/Phoenix to compare against PostgreSQL. Is anybody else here doing this and/or experienced any problems/issues?
I’ll share my results but warning, I’m a noob to it all.
Thanks
Boldly go where no one has ever gone before. ~Star Trek
This is one of those not-so-touched areas and your contribution will be very valuable to the community. I would be watching this thread for getting details about your experiment.
@greenz1 - Here’s what I’m considering doing:
Per the very helpful suggestion of @OvermindDL1 , make use of PostgreSQL’s GIS extension to collect city data from say, Manhattan, via their API endpoint URL, then use FDW foreign data wrapper to allow Spark(Python) to access that data to do it’s “black magic” on it to give me useful analysis (my first time ever doing big data/data science so hopefully Udemy course is good) Also per a suggestion from GIS forum, Python apparently has good analysis libraries like scipy, matplotlib, or scikit-learn to look for patterns so am thinking that is another reason to utilize Spark/Python combination to work with FDW on PostgreSQL
I have no idea if this will work but will give it a shot
Definitely interested in this. It seems like Elixir/Erlang would be the perfect fit for the Lambda Architecture.
Use Bondy ( and optionally Apache Kafka or some other message streaming service) to tie all your microservices together so if you need to do basic analytics or data science in other languages, you can. Use the right tool for the job at the time. Bondy makes it relatively simple to go back and couple in a new implementation.
I would guess they were talking about the BLAS libraries, which are the standard for C-based stuff.
Being complete n00b I am trying to draw some take home conclusions from this thread …
Please correct me if I am wrong.
Yes, more or less. That is my understanding as well.
I have provided some more details on my thoughts in this thread. Maybe it’s useful.
