Elixir vs. Python performance benchmarking

Yeah, like Java. We did a PoC at work and “bare” Java (no Spring, Hibernate, or anything but the bare minimum for external libraries) blew away Elixir, Python, and Go, and came in second only to C.

For our use case Elixir was faster than Python. However, one thing came to light in our PoC that was interesting–Python’s concurrency story is still not great. Async/await code is super hard to reason about and debug. There is very little visibility into the event loop in Python async code, and async code tends to proliferate in your codebase since you can’t call await from a non async function. So when you need to call a coroutine with await the calling function must become async…which means it’s calling function will need to await it and become async. There are ways around this but they are non-obvious and make your code complex.

The Elixir/BEAM approach is so, so much better. Not that it is a silver bullet for complex systems, but if you are writing a large scale system that depends heavily on concurrency for performance, I would choose Elixir over Python.

That’s not even getting into the additional DX issues–introspectability, observability, multi-node deployments, the list goes on.

Python has had a lot of speed optimization go into it over the years. I’m not too surprised that a first solution turned out to be pretty good.

I tried this (building off of the previous performant solution):

    index_file
    |> stream_file()
    |> Task.async_stream(
      fn line ->
        path = :lists.droplast(line)
        {:ok, contents} = :prim_file.read_file(path)
        {:ok, %{"paths" => txt_paths}} = Jason.decode(contents)
        txt_paths
        []
      end,
      max_concurrency: 10,
      ordered: false
    )
    |> Stream.flat_map(fn {:ok, results} -> results end)
    |> Task.async_stream(
      fn path ->
        match?({:ok, _}, :prim_file.read_file_info(path))
      end,
      max_concurrency: 10,
      ordered: false
    )
    |> Stream.run()

I streamed the file with this code (I think I’m probably reinventing wheels here, but it was educational):

  def stream_file(input_file) do
    Stream.resource(
      fn ->
        {:ok, file} = :file.open(input_file, [:raw, :read, read_ahead: 8192])
        file
      end,
      fn file ->
        case :file.read_line(file) do
          {:ok, line} ->
            {[line], file}

          :eof ->
            {:halt, file}
        end
      end,
      fn file -> :file.close(file) end
    )
  end

This performed more or less the same as the other solutions.

Ok so :file.read_line and go through all the lines?

Reading chunks of data (instead of lines) is awkward in this case because each line contains a value. When processing chunks, you have to manually split on newlines and reassemble any values that got split. (At least, I need more coffee before I can come up with a solution to that). Also :file.read/2 returns charlists, and I’m not sure what kind of overhead it would be introducing to convert those back into strings.

Just for the record, that was my Python code and I made no attempts to optimize – I just poked at it for a few minutes until it worked.

Is that so? Only because you are using a python function to call the library function is doesn’t mean there isn’t a native C implementation under the hood.
What about the abomination the python is at this moment in time? Nobody can’t understand at this point if the language is interpreted or compiled anymore because of how many optimizations are in place to make it fast.

like the best Elixir version uses Erlang functions and types

If you are just getting in elixir you might be thinking that using an erlang library is strange and it is the same as calling C code, however this is definitely not true as elixir gets compiled to erlang, so no overhead is involved here.
Moreover if you have access to 2 separate languages and ecosystems without any setup and overhead why not use whats best from both worlds?

This is very clean code (literally, in Bob Martin’s Clean Coding style.)

Is that so? What about concurrency? The elixir solution above is either using tasks or streams and you are showing a solution that can run only in a blocking manner.

I’d be curious how it would do against raw file

index_file
|> :file.open([:raw, :read_ahead])
|> :file.read(1_000_000)
|> ...

I’m interested in Elixir solutions that are not only faster, but also as clean and naive as the Python solution (from above). It doesn’t use any special Python libraries. It doesn’t obviously drop into C code (like the best Elixir version uses Erlang functions and types. ?)

This is very clean code (literally, in Bob Martin’s Clean Coding style.)

from os.path import exists
import json
from time import time

index_file = "tmp/files/index.txt"

existing = []

def vet_files():
    with open(index_file, 'r') as myfile:
        for line in myfile:
            open_json_file(line.rstrip())
            # dict_obj = json.loads(person_data)


def open_json_file(json_file):
    with open(json_file, 'r') as myfile:
        for line in myfile:
            data = json.loads(line)
            files_exist(data['paths'])

def files_exist(paths):
    for p in paths:
        existing.append(exists(p))

if __name__ == "__main__":
    start_time = int(time() * 1000)
    vet_files()
    end_time = int(time() * 1000)
    print(f'Duration: {end_time - start_time} ms')
    print(all(existing))

Thank you all for the continued input. This is interesting! I formalized my repo to use Benchee so I could continue trying out some variants. Here are the results (so far):

Name                        ips        average  deviation         median         99th %
python                     2.27         0.44 s    ±22.35%         0.45 s         0.64 s
:prim_file async           1.38         0.72 s    ±22.75%         0.63 s         1.04 s
Concurrent                 0.63         1.59 s     ±8.90%         1.59 s         1.78 s
Split file                 0.40         2.51 s    ±22.56%         2.35 s         3.30 s
Task.async_stream          0.32         3.16 s    ±21.99%         3.19 s         3.95 s
:prim_file                 0.31         3.26 s    ±41.38%         2.82 s         5.19 s
File                       0.31         3.27 s    ±24.74%         3.48 s         4.00 s
Jsonrs                     0.28         3.54 s    ±20.90%         3.56 s         4.27 s

Comparison:
python                     2.27
:prim_file async           1.38 - 1.64x slower +0.28 s
Concurrent                 0.63 - 3.59x slower +1.14 s
Split file                 0.40 - 5.68x slower +2.07 s
Task.async_stream          0.32 - 7.16x slower +2.72 s
:prim_file                 0.31 - 7.37x slower +2.81 s
File                       0.31 - 7.41x slower +2.83 s
Jsonrs                     0.28 - 8.02x slower +3.10 s

In short, Python is still the fastest. The fastest Elixir solution (so far) is the one that uses Task.async_stream and the :prim_file:

    index_file
    |> File.stream!()
    |> Task.async_stream(fn line ->
      path = String.trim(line)
      {:ok, contents} = :prim_file.read_file(path)
      {:ok, %{"paths" => txt_paths}} = Jason.decode(contents)

      Enum.each(txt_paths, fn p ->
        :prim_file.read_file_info(p)
      end)
    end)
    |> Stream.run()

I tried variants that used EITHER Task.async_stream OR :prim_file, but they didn’t perform as well. Loading the file into memory instead of streaming it also didn’t perform as well. I haven’t been able to get jiffy working, so I gave jsonrs a try, but unfortunately, it performed the worst of these (!!).

What is challenging here is that the solutions have very different performance characteristics. In other words, it’s easy to fall into a hole here, so I’m hoping to identify patterns to avoid. I should probably try coming up with more simplified use-cases, because this one touches on a lot of things: streaming, checking the file system, and JSON decoding.