nightfury17200
Operating on large amount of data
hello all, I want to operate on large amount of data where I pass a query to get some records and then for each record I have to perform a costly operation (creating large number of redis keys(~1000) for each record data and then deleting it) so here I want to use a stream to make the strain less. My code till now is-
Repo.transaction(
fn ->
query
|> Repo.stream(max_rows: @query_batch)
|> Stream.chunk_every(@query_batch)
|> Stream.each(fn x ->
Enum.to_list(x) |> Enum.each(fn y -> delete_from_redis(y) end)
end)
|> Enum.to_list()
end,
timeout: :infinity
)
here in this code when I tried to do -
Stream.each(x, fn y -> delete_from_redis(y) end)
in place of-
Enum.to_list(x) |> Enum.each(fn y -> delete_from_redis(y) end)
my code did not worked, please explain the reason for that also please tell me what can I do to make my current code more efficient?
Marked As Solved
al2o3cr
Stream.each returns a new stream that calls the supplied function as each element is generated. It doesn’t iterate by itself.
Example:
iex(1)> a = [1,2,3]
[1, 2, 3]
iex(2)> stream = Stream.each(a, &IO.inspect/1)
#Stream<[enum: [1, 2, 3], funs: [#Function<38.58486609/1 in Stream.each/2>]]>
iex(3)> Stream.run(stream)
1
2
3
:ok
iex(4)> Enum.to_list(stream)
1
2
3
[1, 2, 3]
Note that in the second line, stream is bound as a Stream struct but nothing has been IO.inspect-ed yet.
Stream.run causes the stream to compute values but discards them and returns :ok, while Enum.to_list accumulates and returns the values.
Also Liked
benwilson512
Your overall stream does, but you’re doing another Stream.each inside a Stream.each and that inner one isn’t doing anything:
|> Stream.each(fn x ->
Stream.each(x, fn y -> delete_from_redis(y) end)
end)
The inner Stream.each should probably be an Enum.each since the x list is already loaded into memory, streaming here doesn’t accomplish anything.
al2o3cr
Re: your specific question:
Stream.each doesn’t run any computation, you have to call a function like Enum.to_list or Stream.run to make Stream do things.
A more general thought: this code returns all the rows from query which neutralizes a lot of the benefit from streaming; Stream.run will do the computation but ignore the result, if that’s your intent.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"