quda
Massive distributed parallel processing of large data sets with Elixir?
Is there a library/framework in Elixir for massive distributed processing of large data sets (BigData) ?
Actually, I need something like Apache Hadoop.
I found some mentions that GenStage+Flow could do it, but I cannot figure it and cannot grasp the “massive distributed parallel processing” in these frameworks. How to spread it on tens on nodes/cluster to perform “massive” MapReduce to flows of +50TB of data?
Are there other Elixir solutions for this ?
PS: For the time being we are doing this big data processing using a (ancient) custom stack developed on PhP/Hadoop, but as we need to migrate the entire back-end to Elixir I have to look for an Elixir (easy to implement, performant, sustainable) solution for this back-end tool.
Most Liked
bdarla
You could find some relevant material in Concurrent Data Processing in Elixir (PragProg).
In addition to Gestate and Flow, the book discusses Broadway. According to the book, Broadway “offers a convenient way to build data-ingestion pipelines that consume events from external message brokers, like RabbitMQ, Apache Kafka, Amazon SQS, and more”.
I have to admit that although I have not read the book myself yet, but it is certainly relevant to your interests.
quda
Quite indeed. This could be a project per se that requires lots of time/resources for study, planning, design, testing etc. and not a side task to complete “by the end of the month”.
Now I have to convince the customer about this. 
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"