sezaru
Hi everyone.
In my system, I have two separated nodes which are connected to each other:
- Node
Ahave access to a database which has a lot of data. - Node
Bdoesn’t have access to that database, so instead it asks nodeAto query that database and send back that data.
This works fine if the amount of data is small, but if there is a lot of data to be sent, it will basically consume all my RAM during the transfer and crash the node with OOM.
If I wanted to do that in the same node, an obvious solution would be to just use streams, But AFAIK I can’t just send a reference to a stream to another node and expect to be able to consume that stream.
For now, what I’m considering doing is using cursor pagination and just do a bunch of requests to the database until it doesn’t return more that, that way I can chunk it and send a controlled amount per request from node A to B.
I was wondering if there is a better way to do that, how do people normally tackle this kind of problems in the Erlang/Elixir world when they want to send huge amount of data between nodes? Maybe there is a version of Stream that works between nodes that I’m not aware?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 5- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
benwilson512
It will be important to quantify what “huge” means here concretely. It is also important to understand how much memory each node has.
LostKobrakai
There’s two parts to
Streamapi. There’s the construction of new lazy enumerables from non enumerable inputs, and there’s the composition of operations on top of existing (lazy) enumerables.You’re trying to use the second part of your node
B, but want to create the input in nodeA. You can however do both things in nodeB, still benefit from a stream api, but have nodeBbe responsible to setup the initial lazy enumerable. E.g. usingStream.transformto query nodeAlazily, like in chunks.You still need a way to lazily query data between node
AandB, where cursor based pagination might be the solution, but you don’t need to stop usingStream.sezaru
Ah, I will need to check how big the structure is, but my VM has 4 GB of ram and the kernel will kill my instance with OOM if I get more than 250 results in a list (The number of rows is not important, since it is the joins I do inside the structure that makes it be so big..)
sezaru
I see, so if I got this right, what you mean is that I can use Stream.resource/3 to create a stream directly in node
Bthat will abstract away multiple queries to nodeAusing cursor pagination to get the data in chunks?sezaru
I just validated that idea, works great! Thanks @LostKobrakai for the suggestion!