kip

kip

ex_cldr Core Team

Its possible the range you want is a little broader than @mudasobwa suggested (depends on your requirements). The nearest I can establish the relevant unicode blocks are:

1F000..1F02F; Mahjong Tiles
1F030..1F09F; Domino Tiles
1F0A0..1F0FF; Playing Cards
1F100..1F1FF; Enclosed Alphanumeric Supplement
1F200..1F2FF; Enclosed Ideographic Supplement
1F300..1F5FF; Miscellaneous Symbols and Pictographs
1F600..1F64F; Emoticons
1F650..1F67F; Ornamental Dingbats
1F680..1F6FF; Transport and Map Symbols
1F700..1F77F; Alchemical Symbols
1F780..1F7FF; Geometric Shapes Extended
1F800..1F8FF; Supplemental Arrows-C
1F900..1F9FF; Supplemental Symbols and Pictographs

So perhaps ~r/[\x{1F000}-\x{1F9FF}]/u would be an alternative.

3
Post #11
9mm

9mm OP

How do I go about showing the actual encoded values in a string? The nightmare I’m facing right now is that some 3rd party service is giving internal server error, and I originally thought it was from emojis but now I’m thinking maybe its some hidden garbage in the string… i want to be able to ‘see’ the invisible characters somehow

Edit: for example like all the invisible space unicode characters, or whatever other weird unicode things could be in there. I know in the past ive encoutered issues with strings that have invisible things I had to delete and re-type and suddenly it works

1
Post #12
9mm

9mm OP

oh nice! thanks

1
Post #13
kip

kip

ex_cldr Core Team

Try String.codepoints/1. Where the code point can’t be decoded to UTF-8 you’ll see a number.

iex> String.codepoints x
["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p",
 "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", ".", ".", ".", ".", "0", "1",
 "2", "3", "4", "5", "6", "7", "8", "9", " ", "不", "極", ",", "物", "片",
 "類", "書", "車", "裡", ...]

If you want to get the numeric value, add <<0>> to the string:

iex> x <> <<0>>
<<97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
  113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 46, 46, 46, 46, 48, 49, 50,
  51, 52, 53, 54, 55, 56, 57, 32, 228, 184, 141, 230, 165, 181, 239, 188, 140,
  ...>>

If you want the raw integers underneath use to_charlist/1:

iex> to_charlist x
[97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 46, 46, 46, 46, 48, 49, 50,
 51, 52, 53, 54, 55, 56, 57, 32, 19981, 26997, 65292, 29289, 29255, 39006,
 26360, 36554, 35041, ...]
5
Post #14
9mm

9mm OP

Wow awesome, is there a way i can see the byteS?

And also, how do i make IO.inspect show the full output

1
Post #15
kip

kip

ex_cldr Core Team 
IEx.configure [inspect: [limit: :infinity]]

in iex will do the trick

Not sure what you mean about bytes → thats what x <> <<0>> is showing you.

2
Post #16
9mm

9mm OP

Thanks on that tip!

And yes, I guess I’m just not used to the syntax. I don’t really work with this stuff often at all, I just know in ruby I encountered this madness before trying to strip garbage characters and they show up in the strong as \uu1234 or something (I cant quite remember), and it was visible when I showed the bytes. I’ll just try one of these methods and hope something stands out! thanks again for this help :smiley:

0
Post #17
kip

kip

ex_cldr Core Team

So far nothing in your strings is “garbage” since its valid UTF-8. If you do String.codepoints/1 then any invalid encoding will show up clearly as a number rather than a string - thats often the most useful for me.

But if you string is valid UTF-8 and you’re getting errors then its likely that the service you’re interacting with doesn’t know how to handle UTF-8 and you’ll need to work out what the expected encoding is.

2
Post #18
9mm

9mm OP

I guess I was thinking of invalid stuff or wack stuff like: U+00A0 NO-BREAK SPACE

1
Post #19
kip

kip

ex_cldr Core Team

I’m a fan of no-break-space since its really useful for number formatting :slight_smile:

2
Post #20