sezaru
How to efficiently send data between nodes?
Hi everyone.
In my system, I have two separated nodes which are connected to each other:
- Node
Ahave access to a database which has a lot of data. - Node
Bdoesn’t have access to that database, so instead it asks nodeAto query that database and send back that data.
This works fine if the amount of data is small, but if there is a lot of data to be sent, it will basically consume all my RAM during the transfer and crash the node with OOM.
If I wanted to do that in the same node, an obvious solution would be to just use streams, But AFAIK I can’t just send a reference to a stream to another node and expect to be able to consume that stream.
For now, what I’m considering doing is using cursor pagination and just do a bunch of requests to the database until it doesn’t return more that, that way I can chunk it and send a controlled amount per request from node A to B.
I was wondering if there is a better way to do that, how do people normally tackle this kind of problems in the Erlang/Elixir world when they want to send huge amount of data between nodes? Maybe there is a version of Stream that works between nodes that I’m not aware?
Marked As Solved
LostKobrakai
There’s two parts to Stream api. There’s the construction of new lazy enumerables from non enumerable inputs, and there’s the composition of operations on top of existing (lazy) enumerables.
You’re trying to use the second part of your node B, but want to create the input in node A. You can however do both things in node B, still benefit from a stream api, but have node B be responsible to setup the initial lazy enumerable. E.g. using Stream.transform to query node A lazily, like in chunks.
You still need a way to lazily query data between node A and B, where cursor based pagination might be the solution, but you don’t need to stop using Stream.
Also Liked
sezaru
I see, so if I got this right, what you mean is that I can use Stream.resource/3 to create a stream directly in node B that will abstract away multiple queries to node A using cursor pagination to get the data in chunks?
Last Post!
sezaru
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"