What's the best way to chunk a stream by an arbitrary non-newline character?

How about?

File.stream!("input_file.txt", [], 1)
|> Stream.chunk_while("", fn x, acc ->
  case x do
    "," -> {:cont, acc, ""}
    _ -> {:cont, acc <> x}
  end
end, fn _ -> {:cont, []} end)
|> Enum.to_list()

Please don’t roll your own CSV parser. You can always use NimbleCSV.parse_stream.

If you are worried about IO performance, you can also pass options to File.stream! that ensure that the API will read-ahead e.g. 512KB of data – this is a very common practice in Erlang/Elixir land when having to ingest data from files.

I’d agree with you on working with CSVs - it’s preferable to coordinate on some highly optimized open source CSV parsers. However, NimbleCSV.parse_stream and NimbleCSV.to_line_stream both expect line-oriented streams, and if the input doesn’t match that format they won’t be able to chunk properly.

The suggestion to increase the number of bytes read by File.stream! is excellent, though - we probably could improve the IO performance of @nallwhy’s solution by streaming larger chunks. Thanks!

That’s not true. to_line_stream is exactly to transform a non-line chunked stream into a line chunked stream, so the result can be used as the input to parse_stream.

Oh, that’s great.

My impression was that to_line_stream processes a stream that is csv/RFC4180 formatted, but which happens to be arbitrarily chunked at non-line-break locations. And this seems to be the default behavior, but I see I could pass a :newlines option to NimbleCSV.define/2 with whatever delimiter is used in my file stream, which will be used by to_line_stream. Thanks!