Suppose I have a input file with a number in each line. Then I want to pipe each of these numbers in a function that will output something. Finally I want to output each of these somethings in a output file. For example:
input.txt = {123, 456, 756} => F(x) = x +1 => output.txt = {124, 457, 757}
I was thinking about creating a File.stream!, then using a Stream.each to apply F(x) on each element, then use some IO function so I can write it to output file. But I’m kinda confused about choosing the best flow to perform this action. I would like to know which is the simplest and most readable way to do it.
I was trying out something in iex to share with you and found this resource, which looks really comprehensive.
Here’s what I did:
nums = File.stream!("nums")
|> Stream.map(&String.trim/1)
|> Stream.filter(&(&1 !== ""))
|> Stream.map(&String.to_integer/1)
|> Stream.map(&(&1 * 123))
|> Stream.map(&Integer.to_string/1)
|> Enum.join("\n")
File.write!("new_file", nums)
At first I did it using File.read
instead of .stream
but both should work.
3 Likes
If you’ve a huge file, the better options are to split them into multiple chunks to avoid single IO bottleneck and use the flow library efficiently to do it in a much faster way.
https://hexdocs.pm/flow/Flow.html
It provides many convenience functions such as windows and trigger points, etc., kind of eases your implementation.
2 Likes
Sounds like an overkill. The Stream
module would do just fine as @pablodavila already commented.
@krasenyp Yes, it is an overkill for a smaller file. As I’ve already mentioned, it works very much efficiently in case of a file sizing in GBs.