almir-neto
I saw that Ecto repo have the function Repo.stream, but I dont know what the steps happens when a trigger the Stream.run. A query like SELECT * FROM table will waits the full scan of database to proced to the next steps or as data arrives will it be sended on to the Elixir client?
Suppose this code, what are happen between the client and the database?
select_all_query
|> Repo.stream()
|> Stream.map(&some_operation_on_data)
|> Stream.run()
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 4 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
al2o3cr
The Redshift docs include a specific warning about using cursors with a large result set:
AFAIK this is a Redshift-specific issue, as the corresponding section of the PG docs doesn’t mention anything similar.
cevado
As @al2o3cr said, it will retrieve chunks, according to the executed query. if there is no filtering/limiting factor on your Stream processing side of the thing, it will end up with all rows in memory at the endo f the stream execution.
functions like
Stream.run/1,Stream.take/2,Stream.filter/2andStream.reject/2should limit or avoid having all rows returned by the query in memory on your application.edit: otherwise, it will protect the database of returning all rows at once with the cursor but will have all rows in the application memory.
almir-neto
You know if to a Redshift database the
Repo.streamcan avoid brings the entire table at once?al2o3cr
Repo.streamuses a cursor on databases where that’s available, so it will retrieve chunks ofmax_rowsrows at a time.