quda
Is there a library/framework in Elixir for massive distributed processing of large data sets (BigData) ?
Actually, I need something like Apache Hadoop.
I found some mentions that GenStage+Flow could do it, but I cannot figure it and cannot grasp the “massive distributed parallel processing” in these frameworks. How to spread it on tens on nodes/cluster to perform “massive” MapReduce to flows of +50TB of data?
Are there other Elixir solutions for this ?
PS: For the time being we are doing this big data processing using a (ancient) custom stack developed on PhP/Hadoop, but as we need to migrate the entire back-end to Elixir I have to look for an Elixir (easy to implement, performant, sustainable) solution for this back-end tool.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 6- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
bdarla
You could find some relevant material in Concurrent Data Processing in Elixir (PragProg).
In addition to Gestate and Flow, the book discusses Broadway. According to the book, Broadway “offers a convenient way to build data-ingestion pipelines that consume events from external message brokers, like RabbitMQ, Apache Kafka, Amazon SQS, and more”.
I have to admit that although I have not read the book myself yet, but it is certainly relevant to your interests.
LostKobrakai
GenStage, Flow, Broadway can be building blocks for something like that, but they‘re certainly not a solution for that problem without a lot of additional work put into the parts not provided by them. Especially the distribution part does not exist in them.
quda
Quite indeed. This could be a project per se that requires lots of time/resources for study, planning, design, testing etc. and not a side task to complete “by the end of the month”.
Now I have to convince the customer about this.
MrDoops
You can also look at GitHub - elixir-explorer/explorer: Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir · GitHub for fast columnar / OLAP workloads - some sort of Broadway ETL setup that grabs a CSV, imports to Explorer, runs your OLAP workloads, then loads the aggregated/transformed results is a very common sort of pipeline Explorer would be good at.
I’d recommend also looking at solutions such as https://flink.apache.org/ or https://materialize.com/ depending on your use case.
Flow + Broadway might get you far enough though if PHP + Hadoop is working already. I have a sneaking suspicion the Elixir ecosystem, maybe via the Nx projects, will end up with some Apache BEAM / distributed dataflow type tooling. At the very least something similar to Explorer taking elixir expressions that execute via Rust NIFs but in streaming dataflow contexts like Apache BEAM (the other BEAM - not our BEAM, but which is also cool).
SirWerto
Maybe you can give a look to Vessel. It is a interface with Hadoop written in Elixir. I don’t know the status of the package but could be helpful if you are keeping Hadoop in your stack.
quda
Vessellooks quite promising for our scope, but sadly it seems abandoned and un-released: “Vessel is currently in a pre-v1 state”. And not a word about distributed processing.