cblavier

cblavier

Converting hex chars to unicode equivalent

Hey there :waving_hand:,

I fetch some HTML content with Req and extract plain text with Floki.

The retrieved text contains hex characters such as "Bient\xf4t" where I would expect "Bientôt" or the unicode equivalent "Bient\u00f4t"

I need the following test to pass:

"Bient\xf4t" |> some_magic() |> String.contains?("Bientôt") == true

Marked As Solved

hauleth

hauleth

In this particular case (where it is Latin1) you can use built in function instead of @kip’s code:

:unicode.characters_to_binary("Bient\xf4t", :latin1)

And you are good to go.

Also Liked

kip

kip

ex_cldr Core Team

\xf4 is a byte representation which Elixir understands, but in this example is not UTF8 encoded, which Elixir does expect. This type of representation happens in some languages when they only support ASCII but need to represent other encodings.

Something like this might help:

defmodule XDecode do
  def decode do
    "Bient\xf4t"
    |> String.chunk(:valid)
    |> decode_codepoints()
  end
  
  def decode_codepoints([utf8, codepoints | rest]) do
    utf8 <> List.to_string(:binary.bin_to_list(codepoints)) <> decode_codepoints(rest)
  end
  
  def decode_codepoints([utf8]) do
    utf8
  end
  
  def decode_codepoints([]) do
    ""
  end
end

As you would expect, this will only work if in fact the hex bytes resolve to valid Unicode code points. Otherwise, as @hauleth says, you need a more generalised converter.

hauleth

hauleth

You need to use some library that provides translation between different codepages (you also need to know what is the codepage of the input). If it is Latin1 then there is some stuff in OTP that can help you, but if it is other page, then you need to find `i

Last Post!

cblavier

cblavier

even better, thanks!

Where Next?

Popular in Questions Top

Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New

Other popular topics Top

baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement