dmitriid
Transforming Unzip.file_stream error
I’m trying to parse contents of a file in zip file. My idea is to use Unzip to stream contents (the unpacked file might be too large for my taste), and transform it on the fly.
However, something’s weird:
Unzip.file_stream(handle, entry_name)
|> Stream.map(fn e -> IO.iodata_to_binary(e) end)
|> Enum.to_list
[
<<208, 161, 208, 190, 208, 186, 208, 190, 208, 187, 208, 190, 208, 178, 44,
208, 161, 208, 181, 209, 128, 208, 179, 208, 181, 208, 185, 44, 208, 144,
208, 189, 208, 176, 209, 130, 208, 190, 208, 187, 209
This works. Replace it with Stream_flat_map or Enum.take(2) or Stream.transform, and
Unzip.file_stream(handle, entry_name)
|> Stream.transform([], fn e, acc -> {e |> IO.iodata_to_binary, acc} end )
|> Stream.run()
** (ErlangError) Erlang error: :data_error
:zlib.inflateEnd_nif(#Reference<0.60592435.1843265537.68376>)
(unzip 0.11.0) lib/unzip.ex:188: anonymous fn/1 in Unzip.decompress/2
(elixir 1.17.0) lib/stream.ex:1039: Stream.do_transform_inner_list/7
(elixir 1.17.0) lib/stream.ex:1038: Stream.do_transform_inner_list/7
(elixir 1.17.0) lib/stream.ex:1055: Stream.do_transform_inner_enum/7
(elixir 1.17.0) lib/stream.ex:690: Stream.run/1
iex:58: (file)
Something fails somewhere, and I can’t for the life of me figure out where ![]()
Most Liked
akash-akya
there are three elements in the resulting stream
What is the issue? I am not sure, but different number of elements might be because of internal buffer of streaming operations. Either way it is still stream of iodata. If you want the whole file as binary you can use Enum.into(<<>>) instead of Enum.to_list() or better
Unzip.file_stream(handle, entry_name)
|> Enum.into(<<>>, &IO.iodata_to_binary/1)
Second issue is for different reason. Unzip checks CRC at the end of file streaming to ensure data integrity, so when you terminate prematurely without reading completely by calling Enum.take(2) it fails because it cannot verify CRC.
For the same reason Enum.take(6) succeeds because there are only 3 chunks and CRC checks out. I explained bit more details here.
I am thinking of adding a flag to skip checking CRC to support cases like this. I haven’t added it sofar since noone requested it, let me know if you need.
dmitriid
Welp. It works as a charm
All I needed to do was just to process the entire stream ![]()
Thank you for helping out and rubber-ducking!
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security









