Jason complains when two emojis go together

[hectorperez]

Hi,

When a text column of my Phoenix/LiveView app includes “”, Jason complains: invalid byte [1].

Each emoji individually is ok ( or ). If they are separated by text it is also fine (whatever​:mrs_claus:t4:), but when they are together () Jason fails. If I manually reload the page, in my browser I see � — but it’s still failing and LiveView reloading continuously.

By the way, String.valid?/1 says that it is ok.

Any idea what might be wrong? Thanks!

[1]

[error] Ranch listener NotesclubWeb.Endpoint.HTTP had connection process started with :cowboy_clear:start_link/4 at #PID<0.5920.0> exit with reason: {%Jason.EncodeError{message: "invalid byte 0xF0 in <<46, 46, 46, 35, 32, 65, 100, 118, 101, 110, 116, 32, 79, 102, 32, 67, 111, 100, 101, 32, 50, 48, 50, 49, 32, 240, 159, 142, 132, 240, 159, 164, 182, 240, 46, 46, 46>>"}, [{Jason, :encode_to_iodata!, 2, [file: 'lib/jason.ex', line: 213, error_info: %{module: Exception}]}, {Phoenix.Socket.V2.JSONSerializer, :encode!, 1, [file: 'lib/phoenix/socket/serializers/v2_json_serializer.ex', line: 70]}, {Phoenix.Socket, :encode_reply, 2, [file: 'lib/phoenix/socket.ex', line: 690]}, {Phoenix.Socket, :handle_in, 4, [file: 'lib/phoenix/socket.ex', line: 602]}, {Phoenix.Endpoint.Cowboy2Handler, :websocket_handle, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 145]}, {:cowboy_websocket, :handler_call, 6, [file: '/Users/hec/code/e/notesclub/deps/cowboy/src/cowboy_websocket.erl', line: 528]}, {:cowboy_http, :loop, 1, [file: '/Users/hec/code/e/notesclub/deps/cowboy/src/cowboy_http.erl', line: 257]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]}

[adamu]

@hectorperez

@hectorperez

and are different:

iex(1)> String.codepoints("🎄🤶🏽")
["🎄", "🤶", "🏽"]
iex(2)> String.codepoints("🎄🤶")
["🎄", "🤶"]

However, I don’t have any problem decoding either of them with Jason, so you must have a problem with how you are encoding unicode somewhere. It looks like something is getting the encoding of U+1F3FD skin-tone modifier wrong.

iex(2)> Jason.decode!(~S{["🎄🤶🏽"]})
["🎄🤶🏽"]
iex(3)> Jason.decode!(~S{["🎄🤶"]})
["🎄🤶"]
iex(4)> Jason.encode(["🎄🤶🏽"])
{:ok, "[\"🎄🤶🏽\"]"}
iex(5)> Jason.encode(["🎄🤶"])
{:ok, "[\"🎄🤶\"]"

Also: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) – Joel on Software

[hectorperez]

Thank you @adamu for the detailed message and the link.

The problem was in a regex that extracted the match’s surroundings and broke the utf-8 characters.