How to read the file and make a list

Questions / Help
1

I read my file in Elixir with the following code
It takes a long time to read the file

File.stream!("big.txt") |> Enum.map(&String.trim/1)

But in Python, I read the file in the fastest possible way with the following code

[line.rstrip() for line in open("big.txt")]

Do you have a better offer?

Thanks

2

not a Stream expert here, but I’d try something like this

File.stream!("big.txt") |> Stream.map(&String.trim/1) |> Enum.to_list()

:thinking: this seems equivalent to your code though…

3

I load in Python in less than 4 seconds
But in Elixir over 500 seconds and more

4

If I understand the python docs correctly it reads in 4kB - 8kB buffers and not individual lines. The elixir code you posted actually reads the file line by line.

2 Likes
5

On a dummy 100MB text file, the use of a 8kB buffer seems ~25x faster than the default line per line behavior on my machine.

File.stream!("big.txt", [:read], 8192) |> Stream.map(&String.trim/1) |> Enum.to_list()
5 Likes
6

Thank you very much dear friends