j4p3

j4p3

What's the best way to chunk a stream by an arbitrary non-newline character?

When streaming input from a file of unknown size, formatted as a single line of comma-separated values, what’s the best way to operate on each comma-separated element in that file?

Example input file input_file.txt:

"AAA","BBB","CCC","DDD"... for a few million characters

Splitting by line:
File.stream!/3 conveniently defaults to separating by :line, but that mode is fixed to splitting on \n or \r\n.

> File.stream!("input_file.txt") |> Enum.to_list()
> ["\"AAA\",\"BBB\",\"CCC\"..."]

Splitting by byte & chunking stream:
File.stream!/3 also accepts a number of bytes, so setting the byte size to 1 (reading the input as raw, not UTF-8) and passing it to Stream.chunk_by/2 constructs something closer to the stream we want:

> File.stream!("input_file.txt", [], 1) |> Stream.chunk_by(&(&1 == ",")) |> Enum.to_list()
> [                      
    ["\"", "A","A","A", "\""],
    [","],
    ["\"", "B","B","B", "\""],
    [","],
    ...
  ]

From this we could probably chain further operations on this stream to filter out the unwanted punctuation and join the desired characters together, but it doesn’t feel like the best way to solve this problem.

What other patterns are there for chunking a stream by an arbitrary character?

Marked As Solved

nallwhy

nallwhy

How about?

File.stream!("input_file.txt", [], 1)
|> Stream.chunk_while("", fn x, acc ->
  case x do
    "," -> {:cont, acc, ""}
    _ -> {:cont, acc <> x}
  end
end, fn _ -> {:cont, []} end)
|> Enum.to_list()

Also Liked

LostKobrakai

LostKobrakai

That’s not true. to_line_stream is exactly to transform a non-line chunked stream into a line chunked stream, so the result can be used as the input to parse_stream.

dimitarvp

dimitarvp

Please don’t roll your own CSV parser. You can always use NimbleCSV.parse_stream.

If you are worried about IO performance, you can also pass options to File.stream! that ensure that the API will read-ahead e.g. 512KB of data – this is a very common practice in Erlang/Elixir land when having to ingest data from files.

Last Post!

j4p3

j4p3

Oh, that’s great.

My impression was that to_line_stream processes a stream that is csv/RFC4180 formatted, but which happens to be arbitrarily chunked at non-line-break locations. And this seems to be the default behavior, but I see I could pass a :newlines option to NimbleCSV.define/2 with whatever delimiter is used in my file stream, which will be used by to_line_stream. Thanks!

Where Next?

Popular in Questions Top

vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New

We're in Beta

About us Mission Statement