Lazy cartesian product

Questions / Help
,
1

Hi all,

This is my first post in this forum, and although I did look, please forgive me if it’s a duplicate (or something very simple, I’m quite new at elixir).

Onto the question: I have two streams which I create with File.Stream!. I want to create the Cartesian product of the two streams, but only until I have 10000 items (the full Cartesian product will be over double that). What I would like to really do is do this lazily, but I can’t seem to come up with an easy way to do that.

Currently, what I’m doing is this

for a <- File.stream!("a.txt"),
  b <- File.stream!("b.txt"),
  do: (
    a = String.trim(a)
    b = String.trim(b)
    {a, b}
  )

However, this seems to work eagerly. Of note is that I also need to apply some transformations to the elements (as exemplified by the String.trim).

Any idea as to how I could accomplish this in a lazy way, maybe with better use of Streams, or zipping them, or some such?

Thank you so much!

3 Likes
2

Welcome @laurazard!

A regular for-comprehension is equivalent to a chain of flat_map with a map at the end. So this comprehension:

iex(2)> for x <- 1..10, y <- 1..10, do: x * y
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 3, 6, 9, 12,
 15, 18, 21, 24, 27, 30, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 5, 10, 15, 20,
 25, 30, 35, 40, 45, 50, ...]

Is equivalent to:

iex(3)> Enum.flat_map(1..10, fn x -> Enum.map(1..10, fn y -> x * y end) end)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 3, 6, 9, 12,
 15, 18, 21, 24, 27, 30, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 5, 10, 15, 20,
 25, 30, 35, 40, 45, 50, ...]

Which means you can make it a stream with:

iex(3)> Stream.flat_map(1..10, fn x -> Stream.map(1..10, fn y -> x * y end) end)

You can use a similar approach in your code.:slight_smile:

13 Likes
3

Hi @josevalim,

Thank you for the comprehensive answer! This is exactly what I need :grinning_face_with_smiling_eyes:

1 Like