Packmatic — On-the-fly Zip Generation

This looks great, thanks for releasing it.

Packmatic 1.0.0 is now released. This release includes ahead-of-time verification enhancements for the Manifest, among other things. Changelog has been updated.

I am just wondering if you were aware of GitHub - ananthakumaran/zstream: An elixir library to write and read ZIP file in a streaming fashion · GitHub when you started the project. Seems like there is a major overlap in features between the two

Yes. It does not do Unicode names, Zip64, or provide easy Plug integration. Not does it seem to have support for URL based Sources or any Custom Source which was recently added. I have seen a few other libraries like these as well.

The reason of building Packmatic was to solve a specific implementation issue outlined in the rationale section (in the README). Nevertheless, availability of these libraries made the task of implementing Packmatic easier. I have noticed that your library is not acknowledged in the README section and I will do so in the next release.

So packmatic can do unicode filenames on win/mac? I recently found out :zip doesn‘t do so.

Thanks for releasing it! I am currently working on a project which requires some zip file processing. :zip works fine at this moment but in the recent future we will be generating zip files on-the-fly with url based remote files.

I feel your library is such a perfect fit. So glad to know it and definitely will give it a try. Thank you for sharing!

I believe it does if I have interpreted the specification correctly. Haven’t had any issues so far with our own use cases.

See this: Packmatic.Field.Local.FileHeader — Packmatic v2.0.0 (first item in Notes) and this: Packmatic.Field.Central.FileHeader — Packmatic v2.0.0 (also first item).

zip64 is a known issue, but I was assuming it does support utf8 filenames (bit 11 of general purpose flag is set)

I had some similar requirements a few months ago:

1. UTF8 file names
2. Stream file input
3. Stream archive output

For our specific use case we needed to generate very large zips of video data without worrying about disk space. The goal was to create a multi part upload in S3 as the zip was created and to avoid disk writes altogether. The goal was to limit memory use to that of the largest video file. This was an unusual enough use case that I created a new library for it: zap.

Due to the specifics of the use case it doesn’t compress at all (though it could quite easily). Through the use of streams it accumulates inputs and periodically emits chunks of output suitable for S3. Since switching to zap we haven’t had any disk space issues.

Edit: I’m sharing in this thread in case others find themselves in a similar situation. Zap has a specific use case that is much narrower than packmatic, but we probably could have made packmatic work for our needs.

I shall add yours to the list as well. For S3 Multipart Uploads my preference would be to use a separate component to accumulate/buffer chunks as there is a limit on number of chunks, and another on size of chunks.

Further, within Packmatic, the URL Source reads in chunks as well (powered by ibrowse), so it could theoretically process source files that do not fit on the host.

The design rationale was driven by user experience. I wanted the download to start instantaneously, and this could only be achieved by not buffering anything at all prior to vending of the stream. Once the download starts the user will wait.