almir-neto
I saw that Ecto repo have the function Repo.stream, but I dont know what the steps happens when a trigger the Stream.run. A query like SELECT * FROM table will waits the full scan of database to proced to the next steps or as data arrives will it be sended on to the Elixir client?
Suppose this code, what are happen between the client and the database?
select_all_query
|> Repo.stream()
|> Stream.map(&some_operation_on_data)
|> Stream.run()
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 4 of 4 Posts
al2o3cr
Repo.streamuses a cursor on databases where that’s available, so it will retrieve chunks ofmax_rowsrows at a time.almir-neto
You know if to a Redshift database the
Repo.streamcan avoid brings the entire table at once?cevado
As @al2o3cr said, it will retrieve chunks, according to the executed query. if there is no filtering/limiting factor on your Stream processing side of the thing, it will end up with all rows in memory at the endo f the stream execution.
functions like
Stream.run/1,Stream.take/2,Stream.filter/2andStream.reject/2should limit or avoid having all rows returned by the query in memory on your application.edit: otherwise, it will protect the database of returning all rows at once with the cursor but will have all rows in the application memory.
al2o3cr
The Redshift docs include a specific warning about using cursors with a large result set:
AFAIK this is a Redshift-specific issue, as the corresponding section of the PG docs doesn’t mention anything similar.