Regular expressions in Elixir with unicode characters

To make it even funnier, erlang re module, Regex relies upon, treats the composed and decomposed unicode characters differently:

iex|1 ▶ :re.run(String.normalize("ö", :nfc),"\\p{Lu}")
#⇒ {:match, [{0, 1}]}
iex|2 ▶ :re.run(String.normalize("ö", :nfd),"\\p{Lu}")
#⇒ {:match, [{1, 1}]}

Note the matched position. I would treat it as a bug in re module, which should probably be addressed.

There is the/u option to enable Unicode support on regex.

With unicode flag:

iex(1)> Regex.match?(~r/\p{Lu}/u, "Ö")
true
iex(2)> Regex.match?(~r/\p{Lu}/u, "ö")
false

Thank you very much! That’s not the default? Performance reason? Can’t remember when I used ASCII the last time

Btw. Wouldn’t it be better to answer that expression with an error than silently returning a wrong answer?

More likely something along the lines of backwards compatibility.

Its not wrong…

If you do not specify the u option, the erlang regex module will interpret the input as latin-1, so a nfd "ö" is "oÌ" <> <<136>> where the second character is uppercase, for an nfc "ö" the regex sees "¶", where the first character is uppercase.

So as you can see the matches are correct.

Thanks for the answer.

But latin as default? 8+3 characters were “correct” too for filenames some time ago

Hope that Elixir will not be the next MS-DOS. Coming from Ruby and hope that after the String handling and this regex default there are not too many things that don’t make any sense and are just “correct” because they have been implemented in a specific way some “hundret” years ago

Just for the statistics: Latin: < 10% of the websites / Unicode: > 60% (numbers from 2012, today probably much higher). So when Elixir was invented almost nobody used latin any more. So the “/u” as default would make much more sense imho.

This feels more like Java (deprecations for decades) than fun to me… - luckily most of the language really is fun to work with.

Sorry, forgot the link to the source: Official Google Blog: Unicode over 60 percent of the web

Latin comes from the underlying erlang module. u is not the default as it is usually not needed but has measurable impact. Also it makes it easier to simply drop to the erlang regex module.

There was a similar question recently, perhaps the derailed discussion towards the end might help you?

Perfect. Thanks. So it’s either like I thought (performance) or just a lack of interest in ~ 80-90% of the people coding out there - that’s why I loved ruby. no “8+3 filenames” … at least >= 1.9 - so maybe there is hope for a future Elixir version too

At least I have a solution and some hints to the reason. Happy and satisfied.

Thanks!

Puzzles me that I haven’t found that issue in my regex unicode query I made before posting this. However: No I know that the community here is very helpful

if you know you don’t need direct unicode matching then there is no need.

Well, it should be in those peculiar cases that a /l or whatever should be specified. I have not worked with Latin-1 text for almost a decade…