almir-neto
I saw that Ecto repo have the function Repo.stream, but I dont know what the steps happens when a trigger the Stream.run. A query like SELECT * FROM table will waits the full scan of database to proced to the next steps or as data arrives will it be sended on to the Elixir client?
Suppose this code, what are happen between the client and the database?
select_all_query
|> Repo.stream()
|> Stream.map(&some_operation_on_data)
|> Stream.run()
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
al2o3cr
Repo.streamuses a cursor on databases where that’s available, so it will retrieve chunks ofmax_rowsrows at a time.almir-neto
You know if to a Redshift database the
Repo.streamcan avoid brings the entire table at once?cevado
As @al2o3cr said, it will retrieve chunks, according to the executed query. if there is no filtering/limiting factor on your Stream processing side of the thing, it will end up with all rows in memory at the endo f the stream execution.
functions like
Stream.run/1,Stream.take/2,Stream.filter/2andStream.reject/2should limit or avoid having all rows returned by the query in memory on your application.edit: otherwise, it will protect the database of returning all rows at once with the cursor but will have all rows in the application memory.
al2o3cr
The Redshift docs include a specific warning about using cursors with a large result set:
AFAIK this is a Redshift-specific issue, as the corresponding section of the PG docs doesn’t mention anything similar.