hello all, I want to operate on large amount of data where I pass a query to get some records and then for each record I have to perform a costly operation (creating large number of redis keys(~1000) for each record data and then deleting it) so here I want to use a stream to make the strain less. My code till now is-
Stream.each doesn’t run any computation, you have to call a function like Enum.to_list or Stream.run to make Stream do things.
A more general thought: this code returns all the rows from query which neutralizes a lot of the benefit from streaming; Stream.run will do the computation but ignore the result, if that’s your intent.
I read about what stream.run do and what the documentation mentioned was-
Runs the given stream.
This is useful when a stream needs to be run, for side effects, and there is no interest in its return result.
to be honest I was not able to make any sense of it hence I tried using stream.each and thought it would be similar to enum.each and would call a function for each value. Can you please explain how stream.run works?
then shouldn’t the code above your reply run here I have added Stream.run. so first I will make chunks of data with Stream.chunk_every(@query_batch) and then pass it to Stream.each for going to each chunk and then for each chunk I will pass it to Stream.each so that I can iterate over the records and then call the required function for each record, and lastly all of this is passed to stream.run() to implement this. Sorry If my understanding is incorrect.