pelopo

pelopo

Read compressed files as string and not bitstring

Hi, I’m reading a file compressed originally with LZMA and ext .txt.xz. I can read it with stream ok, but the output is a bitstring and not a string. How can I convert it to string. The search didn’t give me any meaningful solutions. Thanks

My func

defmodule FileCounter2 do
  def words_per_line!(path) do
    File.stream!(path, [:read, :compressed], :line)
    |> Stream.map(&String.split/1)
    |> Enum.each(&IO.inspect/1)
  end
end
file = "0001008629-99-000012.txt.xz"
FileCounter2.words_per_line!(file)

My output

[
  <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0,
    116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>,
  <<136, 164, 132, 123, 157, 76, 98, 215, 159, 145, 220, 87, 168, 219, 206, 16,
    211, 225, 94, 191, 230, 193, 54, 16, 26, 112, 245, 120, 145, 191, 25, 244>>,
  <<76, 153, 192, 109, 179, 201, 164, 2, 175, 5, 63, 240, 147, 190, 60, 26, 97,
    253, 81, 38, 245, 131, 209, 93, 119, 109, 57>>
]
[
  <<96, 188, 128, 77, 17, 236, 2, 66, 162, 253, 103, 160, 100, 1, 108, 37, 216,
    25, 220, 55, 21, 177, 158, 216, 91, 37, 123, 195, 164, 89, 75, 186, 133, 97,
    66>>,
  <<229, 135,.......]

Most Liked

kip

kip

ex_cldr Core Team

Strings in Elixir are, by definition, UTF-8 encoded.

When you inspect a binary (a binary is a bitstring that is divisible by 8 bits - therefore a String is also a binary encoded as UTF-8), Elixir will output it as a string if it is valid UTF-8. It will output it as a binary if it is not.

Therefore basically your data is not valid UTF-8 and is therefore not a String.

You can chunk the binary by the valid String parts as follows:

iex> binary = <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>
iex> String.chunk(binary, :valid)
[
  <<253>>,
  <<55, 122, 88, 90, 0, 0, 4>>,
  <<230>>,
  <<214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47>>,
  <<229, 163, 224>>,
  ".",
  <<175>>,
  <<14>>,
  <<134>>,
  <<93, 0, 22>>,
  <<233>>
]

That suggests your data is encoded in some very different way to UTF-8.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I think the important thing here is that there’s no singular way to compress a file, rather there are dozens and dozens of different compression algorithms out there. You need to use a program or library that implements the specific decompression algorithm for the file that you have. There are a couple that have a built in implementation inside erlang (and therefore Elixir) zlib — OTP 29.0.2 (erts 17.0.2). I don’t think however that xz compression is the same as gzip.

Where Next?

Popular in Questions Top

siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lists...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New

Other popular topics Top

mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement