cblavier
Converting hex chars to unicode equivalent
Hey there
,
I fetch some HTML content with Req and extract plain text with Floki.
The retrieved text contains hex characters such as "Bient\xf4t" where I would expect "Bientôt" or the unicode equivalent "Bient\u00f4t"
I need the following test to pass:
"Bient\xf4t" |> some_magic() |> String.contains?("Bientôt") == true
Marked As Solved
Also Liked
kip
\xf4 is a byte representation which Elixir understands, but in this example is not UTF8 encoded, which Elixir does expect. This type of representation happens in some languages when they only support ASCII but need to represent other encodings.
Something like this might help:
defmodule XDecode do
def decode do
"Bient\xf4t"
|> String.chunk(:valid)
|> decode_codepoints()
end
def decode_codepoints([utf8, codepoints | rest]) do
utf8 <> List.to_string(:binary.bin_to_list(codepoints)) <> decode_codepoints(rest)
end
def decode_codepoints([utf8]) do
utf8
end
def decode_codepoints([]) do
""
end
end
As you would expect, this will only work if in fact the hex bytes resolve to valid Unicode code points. Otherwise, as @hauleth says, you need a more generalised converter.
hauleth
You need to use some library that provides translation between different codepages (you also need to know what is the codepage of the input). If it is Latin1 then there is some stuff in OTP that can help you, but if it is other page, then you need to find `i
Last Post!
cblavier
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









