How to handle very big enumerables in comprehensions?

Questions / Help
, ,
8

This seems like a good application for streams; they trade additional runtime complexity for memory usage. They can also be useful to skip unnecessary calculations by stopping the stream early, but that’s not applicable since you need every solution.

Here’s a version of your code above that uses Enum.filter all-at-once:

def solutions_enum(puzzle) do
  puzzle
  |> perms_from_lists()
  # in here: all the permutations are in memory
  |> Enum.filter(&sol_okay?(puzzle, &1))
end

and the same code but with streams:

def solutions_enum(puzzle) do
  puzzle
  |> stream_perms_from_lists()
  # in here: only one permutation is in memory at a time
  |> Stream.filter(&sol_okay?(puzzle, &1))
  # this accumulates all the results
  |> Enum.to_list()
end

The new stream_perms_from_lists/1 will involve a bunch of functions from Stream. For some inspiration, here’s a similar-but-different problem (generating permutations) solved with streams:

4 Likes