dmitriid

dmitriid

Transforming Unzip.file_stream error

I’m trying to parse contents of a file in zip file. My idea is to use Unzip to stream contents (the unpacked file might be too large for my taste), and transform it on the fly.

However, something’s weird:

Unzip.file_stream(handle, entry_name) 
|> Stream.map(fn e -> IO.iodata_to_binary(e) end) 
|> Enum.to_list


[
  <<208, 161, 208, 190, 208, 186, 208, 190, 208, 187, 208, 190, 208, 178, 44,
    208, 161, 208, 181, 209, 128, 208, 179, 208, 181, 208, 185, 44, 208, 144,
    208, 189, 208, 176, 209, 130, 208, 190, 208, 187, 209

This works. Replace it with Stream_flat_map or Enum.take(2) or Stream.transform, and

Unzip.file_stream(handle, entry_name) 
|> Stream.transform([], fn e, acc -> {e |> IO.iodata_to_binary, acc} end ) 
|> Stream.run()


** (ErlangError) Erlang error: :data_error
    :zlib.inflateEnd_nif(#Reference<0.60592435.1843265537.68376>)
    (unzip 0.11.0) lib/unzip.ex:188: anonymous fn/1 in Unzip.decompress/2
    (elixir 1.17.0) lib/stream.ex:1039: Stream.do_transform_inner_list/7
    (elixir 1.17.0) lib/stream.ex:1038: Stream.do_transform_inner_list/7
    (elixir 1.17.0) lib/stream.ex:1055: Stream.do_transform_inner_enum/7
    (elixir 1.17.0) lib/stream.ex:690: Stream.run/1
    iex:58: (file)

Something fails somewhere, and I can’t for the life of me figure out where :slight_smile:

Most Liked

akash-akya

akash-akya

there are three elements in the resulting stream

What is the issue? I am not sure, but different number of elements might be because of internal buffer of streaming operations. Either way it is still stream of iodata. If you want the whole file as binary you can use Enum.into(<<>>) instead of Enum.to_list() or better

Unzip.file_stream(handle, entry_name)  
|> Enum.into(<<>>, &IO.iodata_to_binary/1)

Second issue is for different reason. Unzip checks CRC at the end of file streaming to ensure data integrity, so when you terminate prematurely without reading completely by calling Enum.take(2) it fails because it cannot verify CRC.
For the same reason Enum.take(6) succeeds because there are only 3 chunks and CRC checks out. I explained bit more details here.

I am thinking of adding a flag to skip checking CRC to support cases like this. I haven’t added it sofar since noone requested it, let me know if you need.

dmitriid

dmitriid

Welp. It works as a charm :slight_smile: All I needed to do was just to process the entire stream :slight_smile:

Thank you for helping out and rubber-ducking!

Where Next?

Popular in Questions Top

vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54921 245
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement