RooSoft
Process a large gRPC client response as it is being streamed
Hi!
Trying to efficiently parse a 111MB JSON gRPC response that is bound to exponentially grow over time for a while. Expecting it to hit gigabytes mid-long term… talking about the Lightning Network graph straight from a local node. Here’s what it looks like visually…
Currently using this lib: {:grpc, github: "elixir-grpc/grpc"}
As these lines are being written, it seems there’s no way to access that data until it’s been completely parsed and downloaded in RAM. Not sure about that solution’s scalability…
If possible at all, I’d decode the JSON as it is being streamed from the server. That would make this process even possible on the smallest computer such as a raspberry pi, which is super popular hardware used by node operators.
Thanks for any insight on how to achieve this!
Marc
Most Liked
dimitarvp
Just today I found myself wondering the same. There’s Jaxon but I am not sure I can invest the time to rewrite certain code in the projects I work on. Replacing a simple Jason.decode with who knows how many lines that partially ingest a JSON stream can bring about metric tons of complexity.
Maybe a better question for you would be: is there a way to consume the API in a one-JSON-record-per-line fashion?
dimitarvp
Shame the service doesn’t offer alternatives. 
Forking the library is always an option though. If you get desperate, go for that. In my case, I have to parse no more than 20-30 MB for the moment and the API server is slow – sometimes takes 15-20 seconds to deliver data, but our app isn’t real time for 99% of the data so it’s okay.
Hope you can find a solution.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"