Which is the recommended way of reading and writing in a file?

Suppose I have a input file with a number in each line. Then I want to pipe each of these numbers in a function that will output something. Finally I want to output each of these somethings in a output file. For example:

input.txt = {123, 456, 756} => F(x) = x +1 => output.txt = {124, 457, 757}

I was thinking about creating a File.stream!, then using a Stream.each to apply F(x) on each element, then use some IO function so I can write it to output file. But I’m kinda confused about choosing the best flow to perform this action. I would like to know which is the simplest and most readable way to do it.

I was trying out something in iex to share with you and found this resource, which looks really comprehensive.

Here’s what I did:

nums = File.stream!("nums") 
  |> Stream.map(&String.trim/1) 
  |> Stream.filter(&(&1 !== "")) 
  |> Stream.map(&String.to_integer/1)
  |> Stream.map(&(&1 * 123)) 
  |> Stream.map(&Integer.to_string/1) 
  |> Enum.join("\n")

File.write!("new_file", nums)

At first I did it using File.read instead of .stream but both should work.

3 Likes

If you’ve a huge file, the better options are to split them into multiple chunks to avoid single IO bottleneck and use the flow library efficiently to do it in a much faster way.

https://hexdocs.pm/flow/Flow.html

It provides many convenience functions such as windows and trigger points, etc., kind of eases your implementation.

2 Likes

Sounds like an overkill. The Stream module would do just fine as @pablodavila already commented.

@krasenyp Yes, it is an overkill for a smaller file. As I’ve already mentioned, it works very much efficiently in case of a file sizing in GBs.