Hello Elixir community,
I would like to share with you some results of my first 2 months of Elixir learning and ask important question about Elixir language performance as such.
I’ve made some POC application and was able to run it on production-like host (AWS c5.9xl - 96 cores, 70Gb of RAM) and I got about 3-3.5 time worse tps comparing to our current prod metrics. To troubleshoot I just decided make some very rudimentary tests of the main logic which is key in our business - the logic is very simple:
- iterate over files (compressed JSONs or plain CSVs) and
- line by line make some map transformation and
- perform aggregation by applying a sum function for the decimal value on string key
Simple enough, however the only feature of our business is the volume - it is 1 quadrillion (10^15) records per month what I need to account which is ~380 * 10^6 tps. We have such applications which are handling this volume in prod ATM.
For Elixir capability evaluation I took 2 files a plain csv what I need to aggregate (reduce by sum function) - the small (~1000 records) and big (11.7 million records)
Please see snippet of this file
HeaderLength=8
DatasetName=xxxBillingHourly
CreationTimestamp=1561942800000
StartRangeMarker=2019-07-01-01-00-00
EndRangeMarker=2019-07-01-02-00-00
FieldSpec=field-1,field-2,field-3,field-4,field-5,field-6,field-7,field-8,field-9,startTime,endTime,value
KeySpec=field-1,field-2,field-3,field-4,field-5,field-6,field-7,field-8,field-9,startTime,endTime
DataSources=someS3bucket
007931482,abcStore,abcStore,GetPar,GetPar,,us-west-1,,,2018-07-01 01:00:00,2018-07-01 02:00:00,10
016299379,abcEC2,abcEC2,Hours,Gateway,,,arn:aws:someARN,,2018-07-01 01:00:00,2018-07-01 02:00:00,10
018870929,abcLambda,abcLambda,Second,Invoke,,,arn:aws:some-function,,2018-07-01 01:00:00,2018-07-01 02:00:00,59.35000000000002086
.
.
and used Elixir and Java programs to compare with each other (Java is current language we use).
Bellow a listing of two programs which are doing the same things.
Java
@Test
public void konaAggregate() throws IOException {
long start = System.currentTimeMillis();
System.out.println("Start");
String fName = "/home/temp/1/real_kona";
Path totalFilePath = Paths.get(fName + "_java_aggregated");
Stream<Entry<String, BigDecimal>> stream = Files
.lines(Paths.get(fName))
.skip(9)
.map(l -> {
String[] parts = l.split(",");
return Tuple.of(String.join(",", parts[0], parts[1], parts[2], parts[3], parts[4]), new BigDecimal(parts[11]));
})
.collect(
Collectors.groupingBy(
Tuple2::_1,
TreeMap::new,
Collectors.mapping(Tuple2::_2, Collectors.reducing(BigDecimal.ZERO, BigDecimal::add))
)
)
.entrySet()
.stream();
try (BufferedWriter writer = Files.newBufferedWriter(totalFilePath)) {
stream.forEach(line -> {
String record = line.getKey() + "," + line.getValue().toPlainString();
try {
writer.write(record);
writer.newLine();
} catch (IOException e) {
// ignore
}
});
}
long end = System.currentTimeMillis();
System.out.println("Total time: " + (end - start));
}
Elixir
def aggregate_kona() do
IO.puts("Start")
kona = "/home/temp/1/real_kona"
output_path = kona <> "_elixir_aggregated"
start = :os.system_time(:millisecond)
kona
|> File.stream!
|> Stream.map(&String.split(&1, "\n"))
|> Stream.drop(9)
|> Stream.map(&(&1 |> hd))
|> Stream.map(&parse_granular(&1))
|> Enum.reduce(
%{},
fn %{record_key: key, record_value: value}, acc ->
Map.update(acc, key, value, &Decimal.add(&1, value))
end
)
|> Stream.map(&((elem(&1, 0) <> "," <> Decimal.to_string(elem(&1, 1))) <> "\n"))
|> Stream.into(File.stream!(output_path, [:write, :utf8]))
|> Stream.run()
stop = :os.system_time(:millisecond)
IO.puts("Total: #{stop - start}")
end
def parse_granular(record) do
[p,
pr,
cpc,
ut,
op,
_,
_,
_,
_,
_,
_,
value] =
record |> String.split(",")
%{record_key: p <> "," <> pr <> "," <> cpc <> "," <> ut <> "," <> op, record_value: Decimal.new(value)}
end
Let me give you results:
For small file which is 1K records there is no issues and both are (Elixir is faster) latency is 50 milliseconds
But for the big file (11.7 million records and ~2Gb size)
Java gives 16 seconds latency and is able to aggregate file
Elixir was running for ~20 minutes with no produced result and I just terminated an iex session
To get somewhere, I made flow in Elixir even simpler, I commented a reduce part of the flow and made a run. It gave me latency ~115 seconds. Java version for such work coped for ~13 seconds.
So the question to Elixir community, could you review please my task, implementation whether I have any obvious mistakes in Elixir part and also why Elixir’s performance is such that I simply can not accept it?
Thank you.
