pelopo
Read compressed files as string and not bitstring
Hi, I’m reading a file compressed originally with LZMA and ext .txt.xz. I can read it with stream ok, but the output is a bitstring and not a string. How can I convert it to string. The search didn’t give me any meaningful solutions. Thanks
My func
defmodule FileCounter2 do
def words_per_line!(path) do
File.stream!(path, [:read, :compressed], :line)
|> Stream.map(&String.split/1)
|> Enum.each(&IO.inspect/1)
end
end
file = "0001008629-99-000012.txt.xz"
FileCounter2.words_per_line!(file)
My output
[
<<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0,
116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>,
<<136, 164, 132, 123, 157, 76, 98, 215, 159, 145, 220, 87, 168, 219, 206, 16,
211, 225, 94, 191, 230, 193, 54, 16, 26, 112, 245, 120, 145, 191, 25, 244>>,
<<76, 153, 192, 109, 179, 201, 164, 2, 175, 5, 63, 240, 147, 190, 60, 26, 97,
253, 81, 38, 245, 131, 209, 93, 119, 109, 57>>
]
[
<<96, 188, 128, 77, 17, 236, 2, 66, 162, 253, 103, 160, 100, 1, 108, 37, 216,
25, 220, 55, 21, 177, 158, 216, 91, 37, 123, 195, 164, 89, 75, 186, 133, 97,
66>>,
<<229, 135,.......]
Most Liked
kip
Strings in Elixir are, by definition, UTF-8 encoded.
When you inspect a binary (a binary is a bitstring that is divisible by 8 bits - therefore a String is also a binary encoded as UTF-8), Elixir will output it as a string if it is valid UTF-8. It will output it as a binary if it is not.
Therefore basically your data is not valid UTF-8 and is therefore not a String.
You can chunk the binary by the valid String parts as follows:
iex> binary = <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>
iex> String.chunk(binary, :valid)
[
<<253>>,
<<55, 122, 88, 90, 0, 0, 4>>,
<<230>>,
<<214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47>>,
<<229, 163, 224>>,
".",
<<175>>,
<<14>>,
<<134>>,
<<93, 0, 22>>,
<<233>>
]
That suggests your data is encoded in some very different way to UTF-8.
benwilson512
I think the important thing here is that there’s no singular way to compress a file, rather there are dozens and dozens of different compression algorithms out there. You need to use a program or library that implements the specific decompression algorithm for the file that you have. There are a couple that have a built in implementation inside erlang (and therefore Elixir) zlib — OTP 29.0.2 (erts 17.0.2). I don’t think however that xz compression is the same as gzip.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








