Help with performance (file io)

This link appears to be broken. Does anyone know if that blog moved or is archived somewhere?

I encountered a similar slowdown while reading a 1.5 GiB log file (which contained enormous NUL byte sequences due to corruption) line by line in Elixir 1.4.2 and Erlang 19.2 under Linux 3.16, which took 8 hours and 7 minutes to complete!

However, I was able to bring the execution time down to 90 seconds (thereby achieving a massive 320x speedup) by reading the file as a raw byte stream (instead of UTF-8) and with generous caching:

file = "path/to/very/large/file"
io = file |> File.open!(read_ahead: 128 * 1024) # 128 KiB cache
lines = io |> IO.binstream(:line)

See my blog post for the details on the major contributing sources of this solution. Cheers!

One very relevant area with concurrency is when you need to process multiple files. We have to do something a lot like this at cargosense as part of parsing binary data files that get, and I’m able to do several hundred MB/sec of IO on my MBP by leveraging 4-8 files processing concurrently. With streams, this only uses about 10mb of memory.

I’m not gonna say no since I didn’t try it, but I’m not sure we can get a lot from concurrency here since there’s not a lot of processing on each element, so I think the cost of message passing would shadow any possible benefit of concurrency.

But to be honest, I’m fairly pleased with results. On my machine I can read, filter, and write ~500k rows of CSV in about 1 sec, using about 64kb of memory. That seems decent to me

I actually tried some variants of your approaches first, and they failed
I think this one is the fastest because String.split will just forward to :binary.split if the split pattern is a binary, and then the split will be done in C code. But it’s just a guess on why I think String.split works better here.

One thing that I wonder, but didn’t verify is what happens if a 2 byte codepoint ends up being split over two chunks. I suspect it might still work, but didn’t really check it.

In any case, this was a nice exercise. Thanks for writing the post, which motivated me to do some playing on my own. I think together both of us learned a lot in the process

Just to be clear: there’s no process communication happening here. Everything happens in the same process, since you’re working with both files in the raw mode (default for file streams). For clarification, see “Processes and raw files” in File doc1.

This is interesting. Initially I also thought there’s no process involved, but I kept that statement from the original article assuming that even if we don’t spawn an extra process (which happens with File.open in its default not-raw config) then there is still some low-level process within Erlang VM itself that’s responsible for serving file operations in a non-blocking fashion.

But then again, even if that would be true, I can’t really be sure if that could even be classified as “process”. It may be a thread instead and use other means for delivering data to our process than what we used to call “sending messages”. That seems to be the case according to the top part of this doc. So ultimately you’re right It was a leftover from the original article where I’ve clearly mixed up the concepts of interprocess message passing with a tight stream loop that runs on the same process. I’ve fixed it.

Also, I couldn’t sleep over the fact that streamed version was about 3x slower on my machine than the read version (1.8s vs 0.6s). I played with it a bit more and discovered that streaming bytes works much faster than streaming lines.

This is amazing. I’ve tried taking it further, with this and then this, but failed to beat your version. I wonder if it could be made even faster, perhaps by adding some clever concurrency.

Note that I’m splitting input per bytes, so I’m not sure whether this will work correctly with unicode files.

As far as I understood the docs, String.split is UTF-8 aware. Also, you’re using utf8 qualifiers on every pattern match so it looks like it should work with UTF-8 just fine to me. The IO.binwrite doc includes a warning about using it with devices in unicode mode, but that’s not the case here. I’ve added some emoticons to input CSV and they are still there in the output, but I’m not sure that’s an ultimate proof of being UTF-8 compilant

This looks great!

I’m puzzled about one sentence though:

I was wrong assuming that it’s the process communication that puts the biggest overhead here.

Just to be clear: there’s no process communication happening here. Everything happens in the same process, since you’re working with both files in the raw mode (default for file streams). For clarification, see “Processes and raw files” in File doc.

Also, I couldn’t sleep over the fact that streamed version was about 3x slower on my machine than the read version (1.8s vs 0.6s). I played with it a bit more and discovered that streaming bytes works much faster than streaming lines. That led me to the following solution which shaved down the streaming version to ~ 1s:

def main([ filename, "stream" ]) do
  out_file = File.open!(filename <> ".out", [:write, :raw, :delayed_write])

  File.stream!(filename, [], 4000)
  |> Enum.reduce({"", out_file}, &handle_chunk/2)

  File.close(out_file)
end

defp handle_chunk(chunk, {unfinished_line, file}) do
  (unfinished_line <> chunk)
  |> String.split("\n")
  |> process_lines(file)
end

defp process_lines([unfinished_line], file), do: {unfinished_line, file}
defp process_lines([line | rest], file) do
    if filter_line(line) do
      IO.binwrite(file, line <> "\n")
    end
    process_lines(rest, file)
end

Here, I’m taking chunks of 4k (larger chunks didn’t improve perf). When I read the chunk I append it to the unfinished line from the previous chunk. Then I split on newline, and process all except the last element. The last element is unfinished line which I’ll prepend to the next chunk and repeat.

At this point, streaming version is also faster than Ruby (which is ~ 1.8s on my machine).

Note that I’m splitting input per bytes, so I’m not sure whether this will work correctly with unicode files.

Thank you for this write-up! It clearly explains the various ways to slice this problem, explores Elixir/OTP strengths and weaknesses, while also bringing in the Ruby perspective.

Just wanted to give you (and everyone here) a heads up that a heavily rewritten article on Elixir file I/O is up:

Elixir vs Ruby: File I/O performance (updated)

I think it’s worth it to quote here the key conclusion on files and Elixir:

In case of Elixir you can get similar performance if you put streams into proper use (as shown above) or if you go for a read-all-at-once approach. You can also gain a serious performance edge over Ruby if you make use of pattern matching and recursion.

Therefore, it makes most sense to write such scripts from scratch with precise idea about how to put unique Elixir features into use. I can see some serious use cases here that could take benefit from OTP, pattern matching and streaming, like supervisioned CSV import/export workers, Unix daemons or command line tools. Doing blind conversion, like I did in this experiment, makes little sense and doesn’t yield a fair comparison.

Aside from rewritten conclusions, it also includes a much more logical layout of the whole optimization process. And gives credit where the credit is due I hope this time I’ve nailed it and it’ll serve a proper reference for everyone who stumbles upon this problem.

Yeah, the point was to optimize the code algorithmically. Having established that most of the time is spent in the code which decides whether the line should be filtered or not, I tried to reduce the processing there. We don’t need to process the entire line if we’re deciding on the first column only. This had some savings in your csv example, but could have saved a lot more for larger rows. Your trick with checking the last digit is also a great save, since we don’t have to collect all the digits, nor convert to int.

So now, we’re better than Ruby, although it would be interesting to see how Ruby would perform with similar optimizations.

Moreover, the :delayed_write puts the streaming version in the area of Ruby (which I suspect buffers by default). So my non-conclusive conclusion would be that for this contrived microbenchmark we can expect similar performance in both languages (assuming both implementations are tuned), so I don’t see problems with Elixir I/O compared to Ruby (which your otherwise great post seems to imply).