quda

quda

Massive distributed parallel processing of large data sets with Elixir?

Is there a library/framework in Elixir for massive distributed processing of large data sets (BigData) ?
Actually, I need something like Apache Hadoop.
I found some mentions that GenStage+Flow could do it, but I cannot figure it and cannot grasp the “massive distributed parallel processing” in these frameworks. How to spread it on tens on nodes/cluster to perform “massive” MapReduce to flows of +50TB of data?
Are there other Elixir solutions for this ?

PS: For the time being we are doing this big data processing using a (ancient) custom stack developed on PhP/Hadoop, but as we need to migrate the entire back-end to Elixir I have to look for an Elixir (easy to implement, performant, sustainable) solution for this back-end tool.

Most Liked

bdarla

bdarla

You could find some relevant material in Concurrent Data Processing in Elixir (PragProg).
In addition to Gestate and Flow, the book discusses Broadway. According to the book, Broadway “offers a convenient way to build data-ingestion pipelines that consume events from external message brokers, like RabbitMQ, Apache Kafka, Amazon SQS, and more”.
I have to admit that although I have not read the book myself yet, but it is certainly relevant to your interests.

quda

quda

Quite indeed. This could be a project per se that requires lots of time/resources for study, planning, design, testing etc. and not a side task to complete “by the end of the month”.

Now I have to convince the customer about this. :grimacing:

SirWerto

SirWerto

Maybe you can give a look to Vessel. It is a interface with Hadoop written in Elixir. I don’t know the status of the package but could be helpful if you are keeping Hadoop in your stack.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement