If you want to build a system to do all this yourself, Elixir is a great system to build that orchestration layer with. I think Hadoop provides a lot of that orchestration for you in the context of batch processing, but maybe I do not understand your use case. Hadoop distributes tasks to where the data is and/or where compute resources are available. There are retry parameters, scheduling constraints, you name it. You need more processing power, you just bring more nodes up. I can understand not wanting to deal with the accidental complexity that system brings, and maybe your system doesn’t need it, but I think the statement “Java can’t scale batch processes” is indefensible. That is all I’m pointing out.
jeremyjh
OvermindDL1
Don’t forget that the BEAM is a fantastic glue. It can orchestrate nodes running native code (like rust or C++ or python or whatever) with ease, handling failures, restarting, etc…
4
N0x90
If you need something really speedy use an FPGA
1
wanton7
We use Microsoft Orleans a .NET C# distributed virtual actor framework at work. Maybe that’s something you want to checkout if it’s distributed model is something you can work with. C# performance is at least same as Java, maybe faster, plus C# has good async implementation built in using async and await keywords. You can also use F# functional language instead of C# if you use it through Orleankka.
I’ve read or maybe watched a video that Microsoft has used it to distribute IO and CPU bound work.
0
chulkilee
I’m curious which part is actually slow.
- file IO
- string split / parsing
- passing large map across functions (in reduce)
- putting values into large map
- overhead of stream
- …
3
jola
Based on my experience optimizing a similar code snippet (spent lots of hours on it, measuring each part).
- Streaming into lines is very slow compared to reading the entire string directly. Writing a stream is also slower than writing a complete string. Keeping unicode support is slower than not.
- Splitting is fast when you can split 1 big thing, but slow when you split many small things.
- Passing a large map across functions is not something that takes time. As long as it stays within the process nothing gets moved.
- Putting values in large map is very slow (depending on your definition of large). The time it takes grows with the size of the map (unlike ETS which is constant time).
- Overhead of stream is small, but noticeable with a large number of items (like this case). Memory is the deciding factor. If you can spare it, don’t use Stream.
Another thing, causing lots of GC by constantly growing the heap. With a similar use case script I was able to speed it up considerably by setting the initial process heap size to a very large number, which avoids growing the heap a bunch of times. GC in general is costly, if you can avoid it, do. But that’s up to your memory limits.
ps come to my talk at CodeElixir LDN or ElixirConf US where I will be talking about this thing exactly
13
sasajuric
Author of Elixir In Action
In case it wasn’t mentioned (I just took a passing glance at this thread), replacing Enum/Stream functions with plain recursion can sometime improve performance significantly (at the expense of having a more complicated code). Not sure if it would help here, because you seem to be doing a lot in each step, but you can give it a try.
In general , I wouldn’t be surprised if in some scenarios a sequential Java program is about 10x faster than Elixir. BEAM languages are definitely not famous for their speed 
That said, IME, in many cases it is possible to get close enough to the speed of faster languages with a combination of proper algorithms and BEAM specific technical tricks.
4
frumos
Good evening everybody, sorry for delayed update, I had extra hectic working week so put my POC on hold, but today I finally competed first part and got more or less working version.
In short results are just tremendous - I did not expect such a great outcome. And outcome is as following:
In Elixir app:
- each hour, I download ~400 CSV-like files from S3, total size ~70-75Gb. Total # of records to aggregate is ~0.55-0,7 billions. Files are in different size from 0.5Mb to 2,2Gb
- after downloading each files is processed with aggregation function what we discussed here already
the average latency is ~17-18 minutes for the whole work, which is almost 2 times better than I did with Java prototype 1 year ago!!!
The CPU usage is ~50%, the memory usage ~50% which is slightly higher than in Java but totally acceptable (please see screenshot bellow).
I performed 3 test runs for different hours. The test platform is my cloud devbox C5.9XL (36 cores, 25Gb net, 72Gb RAM)
This is first functional step in the POC (and I still need to work a lot) but first results exceed all my expectation!
Thanks to everybody who helped to reach me such results.
28