I read my file in Elixir with the following code
It takes a long time to read the file
File.stream!("big.txt") |> Enum.map(&String.trim/1)
But in Python, I read the file in the fastest possible way with the following code
[line.rstrip() for line in open("big.txt")]
Do you have a better offer?
not a Stream expert here, but I’d try something like this
File.stream!("big.txt") |> Stream.map(&String.trim/1) |> Enum.to_list()
this seems equivalent to your code though…
I load in Python in less than 4 seconds
But in Elixir over 500 seconds and more
If I understand the python docs correctly it reads in 4kB - 8kB buffers and not individual lines. The elixir code you posted actually reads the file line by line.
On a dummy 100MB text file, the use of a 8kB buffer seems ~25x faster than the default line per line behavior on my machine.
File.stream!("big.txt", [:read], 8192) |> Stream.map(&String.trim/1) |> Enum.to_list()
Thank you very much dear friends