preciz
How to properly convert this ?Octal? encoded UTF16 BE string to UTF8
This is the string:
"\\376\\377\\0001\\0009\\000 \\0001\\0002\\000 \\0001\\0009\\000 \\000P\\000r\\000o\\000g\\000r\\000a\\000m\\000m\\000f\\000o\\003\\b\\000r\\000d\\000e\\000r\\000u\\000n\\000g\\000 \\0002\\0000\\0002\\0000\\000 \\000-\\000 \\000A\\000n\\000l\\000a\\000g\\000e\\000 \\0002\\000 \\000-\\000 \\000F\\000o\\003\\b\\000r\\000d\\000e\\000r\\000e\\000r\\000g\\000e\\000b\\000n\\000i\\000s\\000s\\000e\\000.\\000d\\000o\\000c\\000x"
That should be converted to: "19 12 19 Programmförderung 2020 - Anlage 2 - Förderergebnisse.docx"
It’s from a PDFs /Info dictionary.
The "\\376\\377" is a BOM (byte order mark) meaning it’s an UTF16 big endian.
Firefox can read it properly and I assume this is the source code they use:
https://github.com/mozilla/pdf.js/blob/50bc4a18e8c564753365d927d5ec6a6d2cce3072/src/display/metadata.js#L37
I did try to replicate this in Elixir but when it gets to the "ö" it breaks into non relevant characters.
What would be the correct way to convert the string?
Marked As Solved
malaire
That shown code isn’t enough to decode the string as it doesn’t handle decoding \\003\\b to UTF16 codepoint 0x0308.
This could be fixed e.g. by first converting \\b to \\010, but full solution would need a list of all possible escape codes used in the encoding.
Also Liked
malaire
It looks like UTF16 BE bytes in printable ASCII range are represented as ASCII characters (e.g. P instead of \\120), and at least some UTF16 BE bytes in ASCII control character range use the usual escape codes (e.g. \\b meaning ASCII backspace).
So encoding might also be using some other escapes like \\n for \\012 (line feed) or \\r for \\015 (carriage return).
g-andrade
This seems to do it in Erlang:
convert_text_with_bom_to_utf8() ->
Data = <<"\376\377\0001\0009\000 \0001\0002\000 \0001\0009\000 "
"\000P\000r\000o\000g\000r\000a\000m\000m\000f\000o\003"
"\b\000r\000d\000e\000r\000u\000n\000g\000 \0002\0000"
"\0002\0000\000 \000-\000 \000A\000n\000l\000a\000g"
"\000e\000 \0002\000 \000-\000 \000F\000o\003\b\000r\000d\000e\000r"
"\000e\000r\000g\000e\000b\000n\000i\000s\000s\000e"
"\000.\000d\000o\000c\000x"
>>,
{Encoding, BomLength} = unicode:bom_to_encoding(Data), % determine Data encoding from BOM
<<_:BomLength/bytes, Text/bytes>> = Data, % thrown BOM away
<<_/bytes>> = unicode:characters_to_binary(Text, Encoding, utf8). % convert to UTF-8
You can interact with it here.
Unfortunately I wasn’t able to have declare an equivalent literal value in Elixir; but, as long as your string is in the expected format, the same pattern should work.
If, however, the double-backslashes represent actual backslashes in the original string and are not the result of printing / inspecting it, then they’ll need to be unescaped first.
g-andrade
Now, as far as unescaping the string - if needed - this seems to do the trick:
def unescape(data) do
charlist = String.to_charlist(data)
erlang_literal = '"#{charlist}"'
{:ok, [{:string, _, unescaped_charlist}], _} = :erl_scan.string(erlang_literal)
List.to_string(unescaped_charlist)
end
What it does is transform the string into a charlist containing an Erlang “string expression”, which it proceeds to parse - and so I wouldn’t trust it with untrusted input without further investigation.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance










