Hi @jola, did you try getting rid of the :line option and processing chunk by chunk? What I’ve seen so far (more details here: Streaming lines from an enum of chunks) is that this
File.stream!("numbers.txt",[],2048) #chunks instead of lines
|> Stream.transform("",fn chunk, acc ->
[last_line | lines] =
acc <> chunk
|> String.split("\n")
|> Enum.reverse()
{Enum.reverse(lines),last_line}
end)
seems to be 2x faster than using the :line option in File.stream!. I still have to try it in your code and see if there is any difference.
