leifg
Reading directly from a zipped stream
Hey there,
a while ago I implemented an Excel Parser Library. So far it’s working but it has the problem that it doesn’t scale with file size.
Background: The (2000) Excel format is just a bunch of compressed (zip) XML files. In order to get access to the underlying XML, I extract the whole archive in memory:
:zip.extract spreadsheet_filename, [:memory]
This of course is very bad in terms of memory usage. In other languages I have found examples where a compressed archive is directly read as a stream, filtered by filename and then processed.
Is there a way to do this in the Erlang/Elixir world?
Marked As Solved
effinbanjos
A little follow-up many months later:
I finally had a little time to play with this and found that StreamGzip would die after a bit of processing. After casting about for other solutions, I found that the core File.stream! supports compressed files - you just need to feed it the right mode:
File.stream!("./huge-file.gz", [:compressed])
|> Stream.map(&IO.inspect(&1))
|> Stream.run()
File.stream! also defaults to a line-by-line output mode instead of to a byte chunk output mode. I ended-up having to used chunk_while to re-order bytes chunks into line chunks with the StreamGzip library since I wanted to emit individual records into RabbitMQ.
HTH
Also Liked
bbense
Looking at the documentation for the :zip library , I think you can get what you want using the :zip.open interface. Open the handle with the :memory option, list all the files and then use :zip.get
to get each individual file.
You could wrap this with Stream.resource if you want to get all elixiry on it.
leifg
Thanks for the info. Played around with it but I’m not sure this is the way to go.
the :zip.zip_get function always blocks and returns the whole content of the referenced file. What I actually need is a function that returns chunks of the content so that I can process it right away.
Side Note: I peeked at the zip source code in OTP. It seems as if Erlang is reading the contents in chunks. So from research it seems to me that Erlang does not expose this API.
effinbanjos
Did you ever find a solution for this issue? I have a use-case where I’d need to read from a very large compressed file (large for me, that is…22G compressed / 65G+ uncompressed).
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"