Decompressing file with Zstd

MartinElvar · August 16, 2019, 9:54am

I’m trying to decompress a zstd compressed file using the Elixir, and the library https://github.com/chenzhuoyu/elixir-zstd

I’ve tried the following

    {:ok, content} = File.read("myfile")
    <<header :: binary-size(12), remainder :: binary>> = content

    {:ok, orig} = ExZstd.simple_decompress(remainder)

->  ** (MatchError) no match of right hand side value: {:error, 'Unable to determain decompressed size'}

There er no problems when running the zstd command from the terminal.

I’ve also trying the streaming solution, found in the test example.

github.com

chenzhuoyu/elixir-zstd/blob/master/test/ex_zstd_test.exs

defmodule ExZstdTest do
  use ExUnit.Case
  doctest ExZstd

  test "Simple Compression" do
    {:ok, data} = ExZstd.simple_compress("hello, world")
    {:ok, orig} = ExZstd.simple_decompress(data)
    assert orig == "hello, world"
  end

  test "Stream Decompression" do
    zcs = ExZstd.cstream_new()
    ExZstd.cstream_init(zcs)
    {:ok, data} = ExZstd.stream_compress(zcs, "hello, world")
    {:ok, tail} = ExZstd.stream_flush(zcs)

    zds = ExZstd.dstream_new()
    ExZstd.dstream_init(zds)
    {:ok, orig} = ExZstd.stream_decompress(zds, [data, tail])
    assert orig == "hello, world"

This file has been truncated. show original

But i fail to see how i get the tail, when the data is from a file, and not a compression stream.

Any who can help me out, or should i simply resort to calling the zstd command directly from my code.

sribe · August 16, 2019, 2:14pm

I don’t know this library, but I bet you need to pass the whole file contents in rather than discarding the header.

MartinElvar · August 16, 2019, 2:38pm

That is a header that we manually put in, in the the (rust) program that actually produces the file. It only works in the terminal, if i remove the header we wrap it in.

OvermindDL1 · August 16, 2019, 3:07pm

Well with that error it’s failing on line:

github.com

chenzhuoyu/elixir-zstd/blob/b0f045b0d195f116eee317701ee4517bf16ebf57/c_src/nif_ex_zstd.c#L476


      
          {
              size_t ret;
              ErlNifBinary in;
              ErlNifBinary out;
          
              /* extract the binary data */
              if (!(enif_inspect_iolist_as_binary(self, argv[0], &in)))
                  return _zstd_wrap_inval(self);
          
              /* find the output size */
              if (!(ret = ZSTD_getDecompressedSize(in.data, in.size)))
              {
                  enif_release_binary(&in);
                  return _zstd_wrap_string(self, "Unable to determain decompressed size");
              }
          
              /* allocate output binary */
              if (!(enif_alloc_binary(ret, &out)))
              {
                  enif_release_binary(&in);
                  return _zstd_wrap_nomem(self);

Are you sure it is an actual zstd file? Have you tried writing out the file and skipping the header to a file and using the zstd program at that point?

sribe · August 16, 2019, 3:09pm

Ah, ok

MartinElvar · August 16, 2019, 4:55pm

Yes, i’ve have tried writing remainder to a file, and then decompress it with zstd -d myfile.zst no problem. Maybe it’s version mismatch.

OvermindDL1 · August 16, 2019, 5:58pm

It’s possible, it doesn’t look like that library links a built in zstd library but rather just whatever is available on the system. Maybe try a little hoist program to confirm that, like python with ffi for a quick link and test or so?