cjbottaro
Help with performance (file io)
Hello,
Why is this Elixir code:
defmodule Foo do
def run(file_name) do
File.open! file_name, [:read], fn f ->
IO.stream(f, :line) |> Enum.each(&process_line/1)
end
end
defp process_line(line) do
String.rstrip(line) |> String.split(",")
end
end
[ file_name | _ ] = System.argv
Foo.run(file_name)
So much slow than this Ruby code:
def run(file_name)
File.open file_name, "r" do |f|
f.each_line{ |l| process_line(l) }
end
end
def process_line(line)
line.chomp.split(",")
end
run(ARGV[0])
Elixir:
$ time elixir test.exs ../data_gen/posts.csv
real 0m24.496s
user 0m23.527s
sys 0m1.983s
Ruby:
$ time ruby test.rb ../data_gen/posts.csv
real 0m6.556s
user 0m6.444s
sys 0m0.100s
I suspect it’s because I’m not streaming the lines properly?
Thanks for the help,
– C
Marked As Solved
cjbottaro
Friend sent me this:
http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang
Also Liked
karolsluszniak
Just wanted to give you (and everyone here) a heads up that a heavily rewritten article on Elixir file I/O is up:
Elixir vs Ruby: File I/O performance (updated)
I think it’s worth it to quote here the key conclusion on files and Elixir:
In case of Elixir you can get similar performance if you put streams into proper use (as shown above) or if you go for a read-all-at-once approach. You can also gain a serious performance edge over Ruby if you make use of pattern matching and recursion.
Therefore, it makes most sense to write such scripts from scratch with precise idea about how to put unique Elixir features into use. I can see some serious use cases here that could take benefit from OTP, pattern matching and streaming, like supervisioned CSV import/export workers, Unix daemons or command line tools. Doing blind conversion, like I did in this experiment, makes little sense and doesn’t yield a fair comparison.
Aside from rewritten conclusions, it also includes a much more logical layout of the whole optimization process. And gives credit where the credit is due 
I hope this time I’ve nailed it and it’ll serve a proper reference for everyone who stumbles upon this problem.
bbense
Getting fast I/O stream processing in Elixir does require some non-obvious tweaks. Due to the way I/O works on the BEAM (there’s a special process that does I/O and passes results to your process as a message), you want the messages to be as long as possible to avoid overheads.
Doing it line by line is about the slowest way possible.
The tricks I have learned are documented in
https://github.com/bbense/beatwc
But in general the bigger chunks you read and process things in the faster it goes. For a 4 meg csv file I would just slurp the whole thing into memory and operate on the resulting binary.
sunaku
I encountered a similar slowdown while reading a 1.5 GiB log file (which contained enormous NUL byte sequences due to corruption) line by line in Elixir 1.4.2 and Erlang 19.2 under Linux 3.16, which took 8 hours and 7 minutes to complete!
However, I was able to bring the execution time down to 90 seconds (thereby achieving a massive 320x speedup) by reading the file as a raw byte stream (instead of UTF-8) and with generous caching:
file = "path/to/very/large/file"
io = file |> File.open!(read_ahead: 128 * 1024) # 128 KiB cache
lines = io |> IO.binstream(:line)
See my blog post for the details on the major contributing sources of this solution. Cheers!
Last Post!
stevensonmt
This link appears to be broken. Does anyone know if that blog moved or is archived somewhere?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"