Streaming data to an output text file

Hi, I am a newbie to Elixir so please forgive this basic question. I have written a stream function using Stream.unfold that produce a potentially infinite series of numbers. I would like to output these to a text file for offline analysis but I’m having trouble locating the appropriate idiom. This is usually trivial in most other languages but I want to do it in a way that conforms to the ‘functional’ outlook, including appropriate use of the pipe operator.

So far I’ve come up with the fowling code:

Module.my_stream_funct()
|> Stream.take(10)
|> Stream.into(File.stream!(“output.txt”, [:write, :utf8]))
|> Stream.run

I have three main concerns with this approach:

  1. The values are all being concatenated together in the output file, while I would like them to be on separate lines. I think I could use a map function to transform the text, but this feels really clunky.
  2. The introduction of the output file in the middle of the sequence feels a bit procedural, where I am trying to become more functional.
  3. As it stands, the code doesn’t protect from file open errors, which doesn’t feel right.
2 Likes

Welcome!

Your code looks pretty good here, let me see if I can address some of your concerns.

  1. If you want them to be on separate lines you’re gonna need to have new line characters. This really doesn’t look clunky at all.
Module.my_stream_funct()
|> Stream.take(10)
|> Stream.map(&[&1, "\n"])
|> Stream.into(File.stream!("output.txt", [:write, :utf8]))
|> Stream.run
  1. You’re always free to move the output stream creation external to some other function IE
def stream_file(input, output) do
  input
  |> Stream.take(10)
  |> Stream.into(output)
end

Now you can create the whole stream via a completely pure function. Your side effects would then get pushed to wherever you want to call that function from IE:

input = File.stream!("input_file")
output_file = File.stream!("output_file")

stream_file(input, output) |> Stream.run
  1. This is more along the lines of erlang’s “let it crash” philosophy. What you have here is fine. In many cases if the files don’t exist, there’s no right thing to do, just crash cause it won’t succeed anyway. If you have some other thing you want to do you can wrap it in a try block, or if it’s in another process you can monitor that process and handle its failure accordingly.
9 Likes

Done for real now! hahaha sorry for the early reply.

One other point, you generally don’t need to specify the :write or :utf8 flags. Stream.into will handle setting the write flag, and utf8 is also often not necessary if you’re streaming in bytes.

1 Like

Thanks Ben - that’s been really helpful. It’l take me a while to get up the learning curve, but that’s half the fun :smiley:

1 Like

Hmmm, adding the map, without changing the file open gives me:

** (ErlangError) erlang error: :terminated
(stdlib) :io.put_chars(#PID<0.53.0>, :unicode, [-0.43209434920634265, “\n”])
(elixir) lib/stream.ex:417: anonymous fn/5 in Stream.do_into/5
(elixir) lib/stream.ex:1227: anonymous fn/3 in Enumerable.Stream.reduce/3
(elixir) lib/stream.ex:562: anonymous fn/3 in Stream.take/2
(elixir) lib/stream.ex:1198: Stream.do_unfold/4
(elixir) lib/stream.ex:1247: Enumerable.Stream.do_each/4
(elixir) lib/stream.ex:426: Stream.do_into/4
(elixir) lib/stream.ex:494: Stream.run/1

17:58:19.931 [error] Process #PID<0.53.0> raised an exception
** (ArgumentError) argument error
(stdlib) :unicode.characters_to_binary([-0.43209434920634265, “\n”], :unicode, :unicode)
(kernel) file_io_server.erl:409: :file_io_server.put_chars/3
(kernel) file_io_server.erl:178: :file_io_server.server_loop/1
[Finished in 2.0s with exit code 1]

Changing the file open too, gives me:

18:00:51.087 [error] Bad value on output port ‘efile’

** (FunctionClauseError) no function clause matching in anonymous fn/2 in Collectable.File.Stream.into/3
(elixir) lib/file/stream.ex:49: anonymous fn({:error, :badarg}, {:cont, [1.708300256626086, “\n”]}) in Collectable.File.Stream.into/3
(elixir) lib/stream.ex:417: anonymous fn/5 in Stream.do_into/5
(elixir) lib/stream.ex:1227: anonymous fn/3 in Enumerable.Stream.reduce/3
(elixir) lib/stream.ex:562: anonymous fn/3 in Stream.take/2
(elixir) lib/stream.ex:1198: Stream.do_unfold/4
(elixir) lib/stream.ex:1247: Enumerable.Stream.do_each/4
(elixir) lib/stream.ex:426: Stream.do_into/4
(elixir) lib/stream.ex:494: Stream.run/1

[Finished in 2.1s with exit code 1]

Solved it by changing the map function slightly:

Module.my_stream_funct()
|> Stream.take(10)
|> Stream.map(&(inspect(&1) <> "\n"))
|> Stream.into(File.stream!("output.txt", [:write, :utf8]))
|> Stream.run

Simplifying the file open, per your advice :slight_smile:

1 Like

Ah yeah sorry about that, I picked a version of adding the new line that only works if you’ve already got a binary.

I would use to_string btw, inspect is just for debugging purposes.

1 Like