Elixir vs. Python performance benchmarking

I’d be curious what would happen if you piped the file into file.read instead of stream. It looks like Python reads the file completely into memory before processing the lines.

Also you might want to check out genstage which is optimized for creating data pipelines.

*disclaimer: I’m less experienced than the average user on this forum.

Interesting, thanks for sharing. In my case, which is mostly about lists crunching, Elixir is many tens of times faster than Python. With some optimizations the advantage becomes many hundreds of times faster even with 5 times larger amounts of data. For example, an imperative algorithm on a list of ~130k dicts in Python would take me about 20 minutes. Trying to rewrite it functionally and seeing the function just get stuck and never return is what made me switch to Elixir . In Elixir, with an imperative algorithm, I’ve been able to get done with a list of ~500k structs in ~200ms.

Your .exs script is ~1000ms for me, while .py one is ~500ms. Elixir 14.3/OTP 25, Python 3.10.7.

Here is a PR that gets the Elixir performance closer to the python performance, though still slower.

https://github.com/fireproofsocks/ex_vs_py/pull/1

I saw from eprof that after Jason, most time was spent in genservers and cleaning up processes. This is due to the File module opening a new process for every opened file.

➜  ex_vs_py git:(main) ✗ mix profile.eprof scripts/vet_files.exs
Warmup...

Duration: 2009 ms
Duration: 1604 ms

Profile results of #PID<0.212.0>
#                                                          CALLS     %  TIME µS/CALL
Total                                                    3460446 100.0 43300    0.13

:gen_server.call/3                                         64901  2.21  9567    0.15
:erlang.monitor/2                                          64902  2.60 11244    0.17
:file.check_args/1                                        184703  2.70 11683    0.06
File.exists?/2                                             54901  2.91 12604    0.23
Enum."-each/2-lists^foreach/1-0-"/2                        64901  2.93 12698    0.20
:file.read_file_info/2                                     54901  3.31 14329    0.26
:erlang.demonitor/2                                        64902  4.34 18778    0.29
:file.call/2                                               64901  5.90 25540    0.39
:gen.do_call/4                                             64901 10.78 46673    0.72
Jason.Decoder.string/6                                   1322723 21.88 94739    0.07

My PR replaces a lot of these calls with :prim_file which is a nif and does not spawn a process for every file. That also allows the Task.async_stream to help performance; including the async_stream while using the File module just leads to the File genserver being a bottleneck.

Next you might try improving the json performance, perhaps with Eljiffy.

I suspect the stream might be the culprit here, if your file is never that big avoid using stream or read bigger chunks at once.

While it’s useful to look at the perf of the erlang code you’d probably also want to evaluate how much of the time is starting up the beam vm. Not sure which otp applications are started by default with mix, but I recently read that they contribute to a good chunk of the startup time for things running on the beam.

Yeah, but in this benchmark all applications are started before the script is executed

I’ve modified both scripts to return a list of results and Elixir beats python here.

start_time = :erlang.monotonic_time(:millisecond)
index_file = "tmp/files/index.txt"

results =
  index_file
  |> File.stream!()
  |> Task.async_stream(fn line ->
    path = String.trim(line)
    {:ok, contents} = :prim_file.read_file(path)
    %{"paths" => txt_paths} = :jiffy.decode(contents, [:return_maps])
  end, max_concurrency: 16, ordered: false)
  |> Stream.flat_map(fn {:ok, results} -> results end)
  |> Task.async_stream(fn file ->
    match? {:ok, _}, :prim_file.read_file_info(file)
  end, max_concurrency: 8, ordered: false)
  |> Enum.map(fn {:ok, result} -> result end)

end_time = :erlang.monotonic_time(:millisecond)
duration = end_time - start_time
IO.puts("Duration: #{duration} ms")
IO.inspect Enum.all? results

vs

from os.path import exists
import json
from time import time

index_file = "tmp/files/index.txt"

existing = []

def vet_files():
    with open(index_file, 'r') as myfile:
        for line in myfile:
            open_json_file(line.rstrip())
            # dict_obj = json.loads(person_data)


def open_json_file(json_file):
    with open(json_file, 'r') as myfile:
        for line in myfile:
            data = json.loads(line)
            files_exist(data['paths'])

def files_exist(paths):
    for p in paths:
        existing.append(exists(p))

if __name__ == "__main__":
    start_time = int(time() * 1000)
    vet_files()
    end_time = int(time() * 1000)
    print(f'Duration: {end_time - start_time} ms')
    print(all(existing))

And the results are 291ms elixir vs 314ms python.

Some general remarks for guidance:

1. Keep in mind the Erlang VM makes specific trade-offs in relation to high-performance, such as process preemption. It is better to have a predictable system that goes slightly less fast than a fast ones that is unpredictable (or crashes)

2. Streams have lower memory usage at the cost of higher CPU usage. If your goal is to go as fast as you can, not using streams may be better (such as File.read! |> String.split("\n", trim: true))

3. You should see benefits by adding Task.async_stream and similar so you can leverage multi-core

4. I would assume that most of the time is taken by JSON parsing so remember Jason is a pure Elixir package. I assume that the json parsing in Python is most likely done in C. So you may have better results by using something like jiffy (and a more apples to apples comparison)

Most of time is spent on accessing files. I don’t know, but I thought that Erlang team has switched to epoll for prim_file on linux

Default file IO in Erlang is fairly slow. I’d recommend using it with the [:raw] option - it bypasses several layers of abstraction that introduce quite a fair bit of overhead.