cjbottaro
Hello,
Why is this Elixir code:
defmodule Foo do
def run(file_name) do
File.open! file_name, [:read], fn f ->
IO.stream(f, :line) |> Enum.each(&process_line/1)
end
end
defp process_line(line) do
String.rstrip(line) |> String.split(",")
end
end
[ file_name | _ ] = System.argv
Foo.run(file_name)
So much slow than this Ruby code:
def run(file_name)
File.open file_name, "r" do |f|
f.each_line{ |l| process_line(l) }
end
end
def process_line(line)
line.chomp.split(",")
end
run(ARGV[0])
Elixir:
$ time elixir test.exs ../data_gen/posts.csv
real 0m24.496s
user 0m23.527s
sys 0m1.983s
Ruby:
$ time ruby test.rb ../data_gen/posts.csv
real 0m6.556s
user 0m6.444s
sys 0m0.100s
I suspect it’s because I’m not streaming the lines properly?
Thanks for the help,
– C
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Marked As Solved- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
cjbottaro
Friend sent me this:
http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang
Also Liked
karolsluszniak
Just wanted to give you (and everyone here) a heads up that a heavily rewritten article on Elixir file I/O is up:
Elixir vs Ruby: File I/O performance (updated)
I think it’s worth it to quote here the key conclusion on files and Elixir:
Aside from rewritten conclusions, it also includes a much more logical layout of the whole optimization process. And gives credit where the credit is due
I hope this time I’ve nailed it and it’ll serve a proper reference for everyone who stumbles upon this problem.
bbense
Getting fast I/O stream processing in Elixir does require some non-obvious tweaks. Due to the way I/O works on the BEAM (there’s a special process that does I/O and passes results to your process as a message), you want the messages to be as long as possible to avoid overheads.
Doing it line by line is about the slowest way possible.
The tricks I have learned are documented in
https://github.com/bbense/beatwc
But in general the bigger chunks you read and process things in the faster it goes. For a 4 meg csv file I would just slurp the whole thing into memory and operate on the resulting binary.
sunaku
I encountered a similar slowdown while reading a 1.5 GiB log file (which contained enormous NUL byte sequences due to corruption) line by line in Elixir 1.4.2 and Erlang 19.2 under Linux 3.16, which took 8 hours and 7 minutes to complete!
However, I was able to bring the execution time down to 90 seconds (thereby achieving a massive 320x speedup) by reading the file as a raw byte stream (instead of UTF-8) and with generous caching:
See my blog post for the details on the major contributing sources of this solution. Cheers!
Last Post!
stevensonmt
This link appears to be broken. Does anyone know if that blog moved or is archived somewhere?