Flow: Implement Overlapping windows/partitions

Is there a way to process elements in flow in a way that a certain number if elements overlap?

For example, if I have this file:

1
2
3
4
5
6
7
8
9
10

I want to process the element in this way:

[1,2], [2,3], [3, 4], [4,5], [5,6], [6,7], [7,8], [8,9], [9,10]

iex(2)> Stream.chunk([1, 2, 3, 4, 5, 6], 2, 1) |> Enum.to_list
[[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]]

Magic! :slight_smile:

So first pass your File.stream! through Stream.chunk and then pipe that into your Flow.

1 Like

Update as of the end of 2021, as I was looking for the same answer but came across this post in the Google Search Results.

Stream.chunk() was deprecated fully with Elixir 1.7 (1.7 deprecations) in 2018 and replaced with Stream.chunk_every/3,4.

To get the same results, you’d do almost the samething:

iex(7)> Stream.chunk_every([1, 2, 3, 4, 5, 6], 2, 1, :discard) |>  Enum.to_list()
[[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]]

Likewise with Stream.chunk() deprecation, Enum.chunk/2/3/4 was deprecated in favor of Enum.chunk_every/2/3/4.

1 Like