Is this true even for “old style” web development without persistent connections from clients?
I see Elixir as a perfect fit for modern web sites with soft real time features (real time notifications, automatic field completion with server side intervention, chat etc.) since you have a lots of clients with a persistent connection, each of them requiring few work on the server.
But let’s consider “old style” sites where client is just served a dynamic page, cached when possible to reduce database access. And we place it behind a reverse proxy which buffers communications from and to clients (so our Elixir/Phoenix server only communicates with the local proxy, and is not impacted by slow clients). Is this still a problem domain which fits Elixir or a more “raw-speed” approach should be explored?
This in mostly out of curiosity and for better understanding: I’m moving to Elixir/Phoenix because I find them very well thought, well documented and robust, not for performances (which of course is a nice plus but not that important to me).
Try to exchange File.ls!(dir) with elem(:prim_file.list_dir(dir), 1) and see what difference that makes.
Edit: Looking at the code again it is most likely the File.dir? and IO.puts that takes the majority of the time. You can do the same trick with File.dir? although it is a bit more convoluted as no equivalent function exists in Erlang.
Doing file traversals is generally not going to be as efficient in Elixir/Erlang as in other languages. I will explain why.
When you call File.open/2 in Elixir, it doesn’t return a file handler. It returns a process (a lightweight thread of execution) that contains the file handler. But the file handler itself is not even a direct file handler, as you would get in C, but it is an instance of a linkedin driver, which is a piece of code that runs isolated in the VM, that then talks to the file handler.
You may be wondering: why all of this indirection then?
The reason why File.open/2 returns a process is because we can then pass this process around nodes and do file writes across nodes. So for example, I can open up a file on node A, pass that reference to node B, and node B can read/write to that file as if it was in node B, but everything is actually happening in node A. So the reason why we do this is because we favor distribution over raw performance.
What about the linked driver thing though? There are two reasons. First of all, let’s remember that those kind of operations need to be implemented in C or a low-level language for syscalls. And while Erlang provides interoperability with C code, in earlier versions, it was not possible to do an I/O based operation from within the C code. If you did that, you could mess up with the Erlang schedulers that are responsible for concurrency. The second reason is that, if you have C code and there is a bug in that C code, then it can cause a segmentation fault and bring the whole system down, so we prefer to keep our systems running. That led the code to be put in those linked drivers.
Of course all of this adds overhead but the reason we are fine with it is because for our use cases it is most likely that you will find yourself passing a file between nodes than traversing directories as fast as possible, so we focus on the former.
The situation has improved in the latest Erlang/OTP 21 release because the VM added the ability to run I/O blocking C code with something called dirty NIFs, so they recently removed the linked drivers for file operations and that improved performance. But still, most calls in the File module is going through processes and what not. You can actually bypass this process architecture, usually by invoking the :prim_file module or passing a [:raw] option to the File module operations and that typically improves things.
But in a nutshell that’s why it won’t be as fast, because there are many cases where we prefer to focus on features such as distribution and fault tolerance than raw performance.
Btw, regarding CSV processing, did you try the nimble_csv library?
Just as an example, if I rewrite your code to avoid calling FIle.dir? multiple times and instead rely on pattern matching:
def walk(dir) do
with {:ok, dirs} <- File.ls(dir) do
Enum.each(dirs, fn file ->
IO.puts fname = "#{dir}/#{file}"
walk(fname)
end)
end
end
Then it is about 40% faster on my test sample. And if I use :prim_file instead of File so we skip the process and the atomicity guarantees:
def walk(dir) do
with {:ok, dirs} <- :prim_file.list_dir(dir) do
Enum.each(dirs, fn file ->
IO.puts fname = "#{dir}/#{file}"
walk(fname)
end)
end
end
then it is roughly twice faster.
EDIT: Actually, I measured those times using the OS time utility, so that includes the time to boot the VM which is roughly 0.170s in my case. So the gains are a more than 50% once we remove the constant factor.
Thank you all for your suggestions, I just tried them and each resulted in an improvement. At the end the code snippet was three times faster than my first approach.
I’d say my main error was considering it an “easy task” in different languages without realizing that there are no easy tasks when everything is ready to run across different nodes. So it was a comparison between something very simple in Python and something quite complex In Elixir. Definitly not comparable!
Yes I used that library but I was also doing other things which, as resulted from this discussion, weren’t trivial as I thought (like listing files in a directory to choose the CSV to read etc.) I’ll do other tests but I’m pretty sure I was doing the same mistakes of the directory traversal example.
Tank you for the detailed explanation of inner working!
The nice thing is that, if you are attempting to parse multiple CSVs, then that’s a problem you can change to leverage concurrency in a relatively straight-forward fashion, so maybe we can even run faster than the other languages once that is taken into account.
An aside, FreeBSD and BEAM are a great combo in particles dtrace support is excellent. I’m happy to answer any questions there if you need help.
Also your escript probably isn’t really a compiled task; try putting it into a module, compiling that, and timing the execution of the module+function from a running vm. Not only is this a more typical scenario, you can start comparing running 1000 parallel runs vs that of python. It’s going to be very clear that a forked worker uses 100x the memory vs the Elixir one, and with better response times.
An artificial benchmark may not give you practical comparisons vs real world running code. But trying to understand the difference can be very instructive.
Finally you may not realise but this thread has the creator of the language, a core contributor to the VM, and people with a decade of production erlang replying. Getting this level of expertise on a random topic is not unusual on the erlang world. We are very lucky.
PS post your escript and let’s see what we can do with it.
The first version is the one posted in message # 10, but after applying suggestions from @garazdawi and @josevalim the final version is in message #16 (about three times faster).
