Doerge

Doerge

(Syntax Error) invalid or reserved Unicode code point

Hey everybody,

I get a response from a (pretty funky) API, and am trying to decode it with HTTPoison.
It fails with something like this:

{:error,
 %Poison.ParseError{
   data: " { \"config\": ...",
   skip: 86293,
   value: "\\ud83d"
 }}

Toying around with it in IEx also doesn’t work:

iex(1)> tmp = "\ud83d"
** (SyntaxError) iex:1:8: invalid or reserved Unicode code point \u{d83d}. Syntax error after: \u

I’m not really sure how to proceed from here.
How can I clean up my string from invalid code points before decoding it? It’s ok for me to just drop the invalid ones, but I would like to keep the valid ones.

Marked As Solved

al2o3cr

al2o3cr

\ud83d is a Unicode “surrogate pair” character; it’s normally followed by another character of the \uDxxx variety to represent a character above \uFFFF in UTF-16 systems. If there’s one in the text by itself (a “lone surrogate”) that string can’t be represented in UTF-8 at all and is invalid.

Jason can parse these (when they are paired):

iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)22> {:ok, decoded} = Jason.decode("{\"foo\":\"\\uD83D\\uDE04\"}")
{:ok, %{"foo" => "😄"}}

But oddly, doesn’t produce them:

iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)23> {:ok, encoded} = Jason.encode(decoded)
{:ok, "{\"foo\":\"😄\"}"}
iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)24> String.to_charlist(encoded)
[123, 34, 102, 111, 111, 34, 58, 34, 128516, 34, 125]

Also Liked

kip

kip

ex_cldr Core Team

Emoji are definitely part of the Unicode specification.

Doerge

Doerge

Thanks for explaining, and pointing me to Jason! It works perfectly!

Where Next?

Popular in Questions Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <...
New
jaysoifer
Is there a way to rollback a specific migration and only that one ("skipping" all the other ones)? Would mix ecto.rollback -v 2008090...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I fore...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I'm a nov...
New
hariharasudhan94
Lets say i have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => "XX...
New

We're in Beta

About us Mission Statement