RooSoft
Hi!
Trying to efficiently parse a 111MB JSON gRPC response that is bound to exponentially grow over time for a while. Expecting it to hit gigabytes mid-long term… talking about the Lightning Network graph straight from a local node. Here’s what it looks like visually…
Currently using this lib: {:grpc, github: "elixir-grpc/grpc"}
As these lines are being written, it seems there’s no way to access that data until it’s been completely parsed and downloaded in RAM. Not sure about that solution’s scalability…
If possible at all, I’d decode the JSON as it is being streamed from the server. That would make this process even possible on the smallest computer such as a raspberry pi, which is super popular hardware used by node operators.
Thanks for any insight on how to achieve this!
Marc
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
dimitarvp
Just today I found myself wondering the same. There’s Jaxon but I am not sure I can invest the time to rewrite certain code in the projects I work on. Replacing a simple
Jason.decodewith who knows how many lines that partially ingest a JSON stream can bring about metric tons of complexity.Maybe a better question for you would be: is there a way to consume the API in a one-JSON-record-per-line fashion?
RooSoft
I have no control over how the server sends data… it’s sending it in one enormous blob…
The problem also is about reading the data while it’s getting in… not even sure it’s possible with the current grpc lib.
dimitarvp
Shame the service doesn’t offer alternatives.
Forking the library is always an option though. If you get desperate, go for that. In my case, I have to parse no more than 20-30 MB for the moment and the API server is slow – sometimes takes 15-20 seconds to deliver data, but our app isn’t real time for 99% of the data so it’s okay.
Hope you can find a solution.