Passing result of a for loop into Stream

Questions / Help
,
1

Given a function generate_numbers which generates a large amount of numbers.

def generate_numbers() do
    n = 20000000 # a very large number
    for l <- 1..n,
        d <- 1..n,
        h <- 1..n,
        w <- 1..n do
      [ l,  d,  h,  w ]
    end
end

and also an imaginary function to consume the list

def consumer() do
  Stream.each(generate_numbers, fn e ->
     do_something(e)
  end)
  |> Stream.run()
end

The for loop is accumulating all [ l, d, h, w ] which is a big list and consumer has to wait for the whole list to be passed.

How can I make generate_numbers to return/yield each generated [ l, d, h, w ] efficiently and pass it one by one to a caller?

I was thinking I can use into: IO.stream(:stdio, :line) to turn the result into a “stream-able” manner but apparently it only stream the result into standard IO.

2

If you want to generate the list lazily, then you need to use Stream functions to generate the Stream of data.

If you do not care if the list is generated fully (its huge though) then you can simply pass it to Stream.each.

1 Like