Read compressed files as string and not bitstring

pelopo

Hi, I’m reading a file compressed originally with LZMA and ext .txt.xz. I can read it with stream ok, but the output is a bitstring and not a string. How can I convert it to string. The search didn’t give me any meaningful solutions. Thanks

My func

defmodule FileCounter2 do
  def words_per_line!(path) do
    File.stream!(path, [:read, :compressed], :line)
    |> Stream.map(&String.split/1)
    |> Enum.each(&IO.inspect/1)
  end
end
file = "0001008629-99-000012.txt.xz"
FileCounter2.words_per_line!(file)

My output

[
  <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0,
    116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>,
  <<136, 164, 132, 123, 157, 76, 98, 215, 159, 145, 220, 87, 168, 219, 206, 16,
    211, 225, 94, 191, 230, 193, 54, 16, 26, 112, 245, 120, 145, 191, 25, 244>>,
  <<76, 153, 192, 109, 179, 201, 164, 2, 175, 5, 63, 240, 147, 190, 60, 26, 97,
    253, 81, 38, 245, 131, 209, 93, 119, 109, 57>>
]
[
  <<96, 188, 128, 77, 17, 236, 2, 66, 162, 253, 103, 160, 100, 1, 108, 37, 216,
    25, 220, 55, 21, 177, 158, 216, 91, 37, 123, 195, 164, 89, 75, 186, 133, 97,
    66>>,
  <<229, 135,.......]

6 comments

#file #learning-elixir

0 371 6

2024-07-25 14:17:08 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

kip

ex_cldr Core Team

Strings in Elixir are, by definition, UTF-8 encoded.

When you inspect a binary (a binary is a bitstring that is divisible by 8 bits - therefore a String is also a binary encoded as UTF-8), Elixir will output it as a string if it is valid UTF-8. It will output it as a binary if it is not.

Therefore basically your data is not valid UTF-8 and is therefore not a String.

You can chunk the binary by the valid String parts as follows:

iex> binary = <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>
iex> String.chunk(binary, :valid)
[
  <<253>>,
  <<55, 122, 88, 90, 0, 0, 4>>,
  <<230>>,
  <<214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47>>,
  <<229, 163, 224>>,
  ".",
  <<175>>,
  <<14>>,
  <<134>>,
  <<93, 0, 22>>,
  <<233>>
]

That suggests your data is encoded in some very different way to UTF-8.

Post #1

pelopo

Got it, thanks.
I sorted it with the code below that uses an external library, but I thought if the built-in option could read a compressed file , there might be a way for a built-in to decode it too. I thought there might something simpler that the code below

defmodule FileCounter do
  def words_per_line!(path) do
    # Full path to the xz executable
    xz_path = "/opt/homebrew/bin/xz" # Adjust this path based on where xz is installed on your system

    # Use `xz` command to decompress the file content in memory
    {decompressed_content, 0} = System.cmd(xz_path, ["-dc", path])

    # Process the decompressed content
    decompressed_content
    |> String.split("\n", trim: true)
    |> Enum.each(&IO.puts/1)
  end
end

Post #2

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I think the important thing here is that there’s no singular way to compress a file, rather there are dozens and dozens of different compression algorithms out there. You need to use a program or library that implements the specific decompression algorithm for the file that you have. There are a couple that have a built in implementation inside erlang (and therefore Elixir) zlib — OTP 29.0.2 (erts 17.0.2). I don’t think however that xz compression is the same as gzip.

Post #3

pelopo

Thanks. No, the zlib doesn’t deal with LZMA algo. I’ll stick with my hacky code then.

Post #4

dimitarvp

You can still use something better to discover and communicate with CLI programs via @akash-akya’s excellent ex_cmd library.

At least you will not have to hardcode the path, and you might be able to stream from the decompressor as well.

Post #5

pelopo

Thanks, I’ll have to check it a bit later as I’m a very new to Elixir and after a couple of initial tries I can’t seem to get the correct syntax applied to my case.

Post #6

— All posts loaded —