Understanding Elixir/Phoenix performance

What strikes me is that almost every task seems several times slower programmed in Elixir than with Python or Perl. Examples are: traversing a directory and read file modification times, read a CSV file line by line and do some basic processing with them etc.

If possible I’d first like to take a look at your elixir code. Sometimes there are ways to improve it a bit.

Can you link to your code please?

My Elixir code is surely awful since I’m new to functional programming and I understand it’s not fair to make comparison with languages I have use since ages.

Two example of Elixir code i found to perform worse than Python and Perl equivalent are these (not mine):

But my question was more generic: I think that we can assume that Elixir is not born for raw speed and that languages like Python and Perl will usually have an advantage for simple non concurrent tasks.
If it’s not the case than my question is meaningless and can safely be deleted.

I think your original statement that raw speed isn’t Elixir/Erlang’s main advantage is correct. Areas where raw computational power is required you can use a language like Rust and write a NIF so you can utilize it from the BEAM.

I would agree with that, also add in process isolation / supervisors / etc.

Using Enum module is eager, meaning everything is loaded into memory, it would be better to use Stream.

Meanwhile I do not understand why not use Path.wildcard()

iex> Path.wildcard("./**")
["README.md", "_build", "_build/dev", "_build/dev/lib",
 "_build/dev/lib/chess_db", "_build/dev/lib/chess_db/consolidated",
 "_build/dev/lib/chess_db/consolidated/Elixir.Collectable.beam",
...]

# lots of dir and files

And then, if You want just files…

iex> Path.wildcard("./**") |> Enum.reject(&File.dir?/1)

# or

iex> Path.wildcard("./**") |> Stream.reject(&File.dir?/1) |> Enum.take(5)

Yes, but please also not that it is generally faster than perl/python/ruby even for computational tasks. It is slow in comparison to C/Java/Go/Rust type of languages. The “erlang/elixir is slow” quote is thrown around so much that people have started believing it is slow in comparison to any language which is not true.

I don’t believe this is true. If you look at https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fast.html for example erlang comes in somewhere in betweeen and faster than perl and python. And these are tasks which are very unsuitable to do in erlang/elixir.

Can it be slower than perl and python for specific tasks? Of course. The task may be much easier to implement in a mutable language, it may rely on highly optimized underlying code or it is actually done in C.

On the other hand: If you have a problem domain which fits erlang/elixir then it will be fast. That is also the reason go doesn’t have more of an advantage. The computational strength of the language is not as important as its concurrent primitives and handling with IO. And even if go is generally faster when it comes to handling things concurrently the gaps narrows because the “speed” of goroutines vs processes and the underlying scheduling even things out.

Web servers are fast in Elixir because web servers don’t do anything most of the time - most of the time they are just waiting. Either for request data or for database, etc. Elixir/Erlang are excellent at finding things to do when one of the processes doesn’t do anything, which makes them generally fast at web servers.

There’s also a question of algorithms. If you use an algorithm designed with mutable data structures in mind, it will be unavoidably slower when used with immutable data structures - on the other hand, there are some algorithms designed for immutable data structures and different, more specialised structures that can shine in some cases.

Finally, there’s the matter of the VM. BEAM is just a very well implemented and a very efficient machine. The runtime system responsible for IO interaction, scheduling and similar things have been optimised over the years. Yes, it does not have a JIT, but the normal emulator is quite fast compared to other VMs. It’s also one of the few register-based VMs in the wide usage, and register VMs generally tend to be faster than the more popular and simpler stack-based VMs.

Thank you everybody!

It seems that while I was right in thinking that features like concurrency are much more important than raw-speed in typical Elixir use-cases, I vastly underestimated the importance of code optimization.

I will try, as an exercise, to rewrite some of the algorithms like the directory traversal example above to make them more efficient (even if in a real word application this would probably be useless).

I suspect that quote is in reaction to all the “Elixir is lightning fast” quotes, which of course are only true when comparing to slow interpreted languages, Ruby in particular. It’s not fast when compared to compiled C++/Rust/Go and Java (well, mostly). But to have the expressiveness of Ruby plus some, at a performance level solidly between Ruby & C, is an awesome win.

Ok, I tried your suggestions for code optimization and other approaches to directory traversal; the fastest way to recursively walk a directory and print file names (I gave up to printing file modify dates to keep things simple) was:

defp walk(dir) do
  Enum.each(File.ls!(dir), fn file ->
    IO.puts fname = "#{dir}/#{file}"
    if File.dir?(fname), do: walk(fname)
  end)
end

walk("/path/to/dir")

I compiled it in an executable with escript and redirected output to avoid measuring terminal speed! On a directory tree with 62.000 files it takes a time variable between 8.81 and 11.17 seconds.

The same task with python required 1.10 → 1.20 seconds.

For comparison, a shell script with find and xargs took 0.73 → 0.76 seconds

I’m wondering if I’m completely missing something obvious or if my platform, FreeBSD, is the problem. I tendo to exclude the latter since I’ve never heard about issues with the Erlang/FreeBSD combination.