cblavier

cblavier

Converting hex chars to unicode equivalent

Hey there :waving_hand:,

I fetch some HTML content with Req and extract plain text with Floki.

The retrieved text contains hex characters such as "Bient\xf4t" where I would expect "Bientôt" or the unicode equivalent "Bient\u00f4t"

I need the following test to pass:

"Bient\xf4t" |> some_magic() |> String.contains?("Bientôt") == true

Marked As Solved

hauleth

hauleth

In this particular case (where it is Latin1) you can use built in function instead of @kip’s code:

:unicode.characters_to_binary("Bient\xf4t", :latin1)

And you are good to go.

Also Liked

kip

kip

ex_cldr Core Team

\xf4 is a byte representation which Elixir understands, but in this example is not UTF8 encoded, which Elixir does expect. This type of representation happens in some languages when they only support ASCII but need to represent other encodings.

Something like this might help:

defmodule XDecode do
  def decode do
    "Bient\xf4t"
    |> String.chunk(:valid)
    |> decode_codepoints()
  end
  
  def decode_codepoints([utf8, codepoints | rest]) do
    utf8 <> List.to_string(:binary.bin_to_list(codepoints)) <> decode_codepoints(rest)
  end
  
  def decode_codepoints([utf8]) do
    utf8
  end
  
  def decode_codepoints([]) do
    ""
  end
end

As you would expect, this will only work if in fact the hex bytes resolve to valid Unicode code points. Otherwise, as @hauleth says, you need a more generalised converter.

hauleth

hauleth

You need to use some library that provides translation between different codepages (you also need to know what is the codepage of the input). If it is Latin1 then there is some stuff in OTP that can help you, but if it is other page, then you need to find `i

Last Post!

cblavier

cblavier

even better, thanks!

Where Next?

Popular in Questions Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 131117 1222
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 55125 245
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement