Iterating over a file stream

Questions / Help
21

That looks like a better solution for my current problem. But what exactly is the difference between

|> Stream.chunk_every(2000, 2000, [])

and

|> Stream.chunk_every(2000)

From the documentation

Streams the enumerable in chunks, containing count items each, where each new
chunk starts step elements into the enumerable.

step is optional and, if not passed, defaults to count, i.e. chunks do not
overlap.

So here count == step. So both are ideally the same I guess.

22

Guys great thanks for the help. I think I should dig more into streams and async tasks which I haven’t done yet.

23

The difference is that |> Stream.chunk_every(2000) will throw away values if you have less than 2000 in the last bucket.

3 Likes
24

Are you sure?

iex(1)> 1..5 |> Stream.chunk_every(2) |> Enum.to_list()
[[1, 2], [3, 4], [5]]
2 Likes
25

Stream.chunk_by (and Enum.chunk_by) would drop “leftover” values, but they’re been deprecated in favor of chunk_every. I’m glad that change was made.

5 Likes
26

To discard we have to pass it as follows it seems.

Stream.chunk_every(2, 2, :discard) |> Enum.to_list()

[[1, 2], [3, 4]]

27

Ahh thanks @gregvaughn I thought I was going crazy. I was remembering the deprecated chunk_by functions. Looks like chunk_every works differently.

4 Likes