preciz

preciz

How to properly convert this ?Octal? encoded UTF16 BE string to UTF8

This is the string:

"\\376\\377\\0001\\0009\\000 \\0001\\0002\\000 \\0001\\0009\\000 \\000P\\000r\\000o\\000g\\000r\\000a\\000m\\000m\\000f\\000o\\003\\b\\000r\\000d\\000e\\000r\\000u\\000n\\000g\\000 \\0002\\0000\\0002\\0000\\000 \\000-\\000 \\000A\\000n\\000l\\000a\\000g\\000e\\000 \\0002\\000 \\000-\\000 \\000F\\000o\\003\\b\\000r\\000d\\000e\\000r\\000e\\000r\\000g\\000e\\000b\\000n\\000i\\000s\\000s\\000e\\000.\\000d\\000o\\000c\\000x"

That should be converted to: "19 12 19 Programmförderung 2020 - Anlage 2 - Förderergebnisse.docx"

It’s from a PDFs /Info dictionary.

The "\\376\\377" is a BOM (byte order mark) meaning it’s an UTF16 big endian.

Firefox can read it properly and I assume this is the source code they use:
https://github.com/mozilla/pdf.js/blob/50bc4a18e8c564753365d927d5ec6a6d2cce3072/src/display/metadata.js#L37

I did try to replicate this in Elixir but when it gets to the "ö" it breaks into non relevant characters.

What would be the correct way to convert the string?

Marked As Solved

malaire

malaire

That shown code isn’t enough to decode the string as it doesn’t handle decoding \\003\\b to UTF16 codepoint 0x0308.

This could be fixed e.g. by first converting \\b to \\010, but full solution would need a list of all possible escape codes used in the encoding.

Also Liked

malaire

malaire

It looks like UTF16 BE bytes in printable ASCII range are represented as ASCII characters (e.g. P instead of \\120), and at least some UTF16 BE bytes in ASCII control character range use the usual escape codes (e.g. \\b meaning ASCII backspace).

So encoding might also be using some other escapes like \\n for \\012 (line feed) or \\r for \\015 (carriage return).

g-andrade

g-andrade

This seems to do it in Erlang:

convert_text_with_bom_to_utf8() ->
    Data = <<"\376\377\0001\0009\000 \0001\0002\000 \0001\0009\000 "
             "\000P\000r\000o\000g\000r\000a\000m\000m\000f\000o\003"
             "\b\000r\000d\000e\000r\000u\000n\000g\000 \0002\0000"
             "\0002\0000\000 \000-\000 \000A\000n\000l\000a\000g"
             "\000e\000 \0002\000 \000-\000 \000F\000o\003\b\000r\000d\000e\000r"
             "\000e\000r\000g\000e\000b\000n\000i\000s\000s\000e"
             "\000.\000d\000o\000c\000x"
           >>,

    {Encoding, BomLength} = unicode:bom_to_encoding(Data), % determine Data encoding from BOM
    <<_:BomLength/bytes, Text/bytes>> = Data, % thrown BOM away
    <<_/bytes>> = unicode:characters_to_binary(Text, Encoding, utf8). % convert to UTF-8

You can interact with it here.

Unfortunately I wasn’t able to have declare an equivalent literal value in Elixir; but, as long as your string is in the expected format, the same pattern should work.

If, however, the double-backslashes represent actual backslashes in the original string and are not the result of printing / inspecting it, then they’ll need to be unescaped first.

g-andrade

g-andrade

Now, as far as unescaping the string - if needed - this seems to do the trick:

  def unescape(data) do                                                                                                                                                                                                                        
    charlist = String.to_charlist(data)                                                                                                                                                                                                        
    erlang_literal = '"#{charlist}"'                                                                                                                                                                                                           
    {:ok, [{:string, _, unescaped_charlist}], _} = :erl_scan.string(erlang_literal)                                                                                                                                                            
    List.to_string(unescaped_charlist)                                                                                                                                                                                                         
  end  

What it does is transform the string into a charlist containing an Erlang “string expression”, which it proceeds to parse - and so I wouldn’t trust it with untrusted input without further investigation.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call t...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42920 311
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39297 209
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" =&gt; #BSON.ObjectId&lt;58eb1a7a9ad169198c3dXXXX&gt;, "email" =&gt; ...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement