Help converting regex (to remove emojis)

9mm

I’m trying to make a regex to remove emojis based on this thread:

This causes an error:

Regex.replace(~r/[\u{1F600}-\u{1F6FF}]/, "💰 Monies! 💲", "")
# (Regex.CompileError) PCRE does not support \L, \l, \N{name}, \U, or \u at position 2

I tried using \x{FFFF} syntax instead according to this: Regex Tutorial: Introduction to Unicode Regular Expressions

But I get another error:

Regex.replace(~r/[\x{1F600}-\x{1F6FF}]/, "💰 Monies! 💲", "")
# (Regex.CompileError) character value in \x{} or \o{} is too large at position 9

Apparently I also need u to enable unicode but that doesnt seem to work either:

Regex.replace(~r/[\x{1F600}-\x{1F6FF}]/u, "💰 Monies! 💲", "")
"💰 Monies! 💲"

23 comments

#regex

2 5987 23

2023-04-12 20:00:42 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

kip

ex_cldr Core Team

You might have some better luck using the unicode character class So (other symbol). For example:

iex> x = "💰 Monies! 💲"                                
"💰 Monies! 💲"
iex> String.replace x, ~r/\p{So}/u, ""
" Monies! "

BTW, the “correct” approach to this would be:

iex> String.replace x, ~r/\p{Emoji}/u, ""

But erlang’s re module doesn’t support that character class.

Post #1

9mm

Interesting.. does that strip other languages or only emojis? I was trying to avoid stripping chinese/japanese/korean, which it seems like a lot of emoji regex do

Post #2

kip

ex_cldr Core Team

No, it doesn’t. The full list of code points in So are here: Unicode Characters in the 'Symbol, Other' Category

It’s more permissive than just emoji, but it does not include scripts.

iex> String.replace "御言宣", ~r/\p{So}/u, ""
"御言宣"

Post #3

9mm

Hmm yess, and it also is missing some emoji for some reason. God this is confusing

Post #4

9mm

Can you try this string?

x = "abcdefghijklmnopqrstuvwxyz....0123456789 不極，物片類書車裡！十今果半接國先雄 ニッポン」「ニホン」両方使用される中 には文중국, 일본, 베트남 등 한자 문화권에 속하는 아시아 여러 국가에서는 한국어的差异外，通常认为还存在词汇上的差异。例如繁体中文里多用的“原 لمنطقة الشرق الأوسط هيلي: التحرك ضد إيران سيبدأ من مجلس الأمن📌☮️💟🔯☪️㊗️🈵🆚💯❕🔞🚷🔰⁉️⚠️💤🌐🌀▶️🔠🔣↔️↩️👁‍🗨🗨🗨🗨🗨 ◽️🔲🇵🇦🏳️🏳️‍🌈🌈🌈🌈🌈🇹🇲🇹🇷🤛🤜🏼👍🏽👌☝🏼🥝🥦🌶🌽🍎 🍲🍔🥞🍝🍔🍗🌮🍯🥠🥢🍴🥄🥂☕️😀😃😄🤣😂😅😆☺️😊😍😌 😘😗😙😚😜😝😛😋🤨🧐🤓😒😏🤩🤩😎😞😔😖😢😣☹️😩🙁🤯 😰😓😦😲🤒🤕👿👹👽✊🏼"

It leaves the other languages properly but some emojis are left.

Post #5

9mm

I did find this guys ruby gem but I just wasted like an hour trying to convert this regex and when it finally worked it erased all the chinese characters, so obviously I got that wrong

https://github.com/guanting112/remove_emoji

Post #6

mudasobwa

Creator of Cure

There are many issues here. Both characters above are not in the range you specified in the first place.

▶ to_charlist "💰" 
#⇒ [128176]
▶ to_charlist "💲"
#⇒ [128178]
▶ 0x1F600
#⇒ 128512

To make it work for the range with \u, just interpolate the literals (I voluntarily changed the starting value for the range to what it probably should be):

▶ Regex.replace(~r/[#{"\u{1F000}"}-#{"\u{1F6FF}"}]/u, "💰 Monies! 💲", "")
" Monies! "

\x will work out of the box with a proper range:

Regex.replace(~r/[\x{1F000}-\x{1F6FF}]/u, "💰 Monies! 💲", "")
" Monies! "

Post #7

kip

ex_cldr Core Team

Yes, I see the same. It’s a bit surprising since according to some utility code I wrote they look to be So but re certainly doesn’t agree:

iex> x
"🤨🧐🤓🤩"
iex> Cldr.Unicode.Category.category x 
[:So, :So, :So, :So]
iex> String.replace x, ~r/\p{So}/u, ""
"🤨🧐🤓🤩"

Post #8

kip

ex_cldr Core Team

Seems that emoji belong to the Common script so this appears to get closer - but it also deletes digits and punctuation.

iex> x = "abcdefghijklmnopqrstuvwxyz....0123456789 不極，物片類書車裡！十今果半接國先雄 ニッポン」「ニホン」両方使用される中 には文중국, 일본, 베트남 등 한자 문화권에 속하는 아시아 여러 국가에서는 한국어的差异外，通常认为还存在词汇上的差异。例如繁体中文里多用的“原 لمنطقة الشرق الأوسط هيلي: التحرك ضد إيران سيبدأ من مجلس الأمن📌☮️💟🔯☪️㊗️🈵🆚💯❕🔞🚷🔰⁉️⚠️💤🌐🌀▶️🔠🔣↔️↩️👁‍🗨🗨🗨🗨🗨 ◽️🔲🇵🇦🏳️🏳️‍🌈🌈🌈🌈🌈🇹🇲🇹🇷🤛🤜🏼👍🏽👌☝🏼🥝🥦🌶🌽🍎 🍲🍔🥞🍝🍔🍗🌮🍯🥠🥢🍴🥄🥂☕️😀😃😄🤣😂😅😆☺️😊😍😌 😘😗😙😚😜😝😛😋🤨🧐🤓😒😏🤩🤩😎😞😔😖😢😣☹️😩🙁🤯 😰😓😦😲🤒🤕👿👹👽✊🏼"
"abcdefghijklmnopqrstuvwxyz....0123456789 不極，物片類書車裡！十今果半接國先雄 ニッポン」「ニホン」両方使用される中 には文중국, 일본, 베트남 등 한자 문화권에 속하는 아시아 여러 국가에서는 한국어的差异外，通常认为还存在词汇上的差异。例如繁体中文里多用的“原 لمنطقة الشرق الأوسط هيلي: التحرك ضد إيران سيبدأ من مجلس الأمن📌☮️💟🔯☪️㊗️🈵🆚💯❕🔞🚷🔰⁉️⚠️💤🌐🌀▶️🔠🔣↔️↩️👁‍🗨🗨🗨🗨🗨 ◽️🔲🇵🇦🏳️🏳️‍🌈🌈🌈🌈🌈🇹🇲🇹🇷🤛🤜🏼👍🏽👌☝🏼🥝🥦🌶🌽🍎 🍲🍔🥞🍝🍔🍗🌮🍯🥠🥢🍴🥄🥂☕️😀😃😄🤣😂😅😆☺️😊😍😌 😘😗😙😚😜😝😛😋🤨🧐🤓😒😏🤩🤩😎😞😔😖😢😣☹️😩🙁🤯 😰😓😦😲🤒🤕👿👹👽✊🏼"
iex> String.replace x, ~r/\p{Common}/u, ""
"abcdefghijklmnopqrstuvwxyz不極物片類書車裡十今果半接國先雄ニッポンニホン両方使用される中には文중국일본베트남등한자문화권에속하는아 시아여러국가에서는한국어的差异外通常认为还存在词汇上的差异例如繁体中文里多用的原لمنطقةالشرقالأوسطهيليالتحركضدإيرانسيبدأمنمجلسالأمن️️️️️️️️‍️️️‍️️️"

Post #9