abtrapp

abtrapp

Regular expressions in Elixir with unicode characters

Hi!

Can somebody please explain this:

iex(9)> Regex.match?(~r/\p{Lu}/, "Ö")
true
iex(10)> Regex.match?(~r/\p{Lu}/, "ö")
true
iex(11)> Regex.match?(~r/\p{Lu}/, "o")
false
iex(12)> Regex.match?(~r/\p{Lu}/, "O")
true

Why are all unicode characters that are not from the english alphabet uppercase characters in Elixir (in contrast to every other language I know and the definition of \p{Lu})?

Elixir 1.7.4 (compiled with Erlang/OTP 21)

Marked As Solved

chriseyre

chriseyre

There is the/u option to enable Unicode support on regex.

Also Liked

michalmuskala

michalmuskala

I will agree that the elixir regexes should include the u flag by default and require explicit opt-out. We generally should prefer correctness over speed with ways to opt-in into faster, but potentially incorrect results.

Unfortunately, the soonest we could be able to change this would be 2.0 (and there are no plans for 2.0 at the moment). This is because that would be a breaking change since many functions would start returning different results.

kip

kip

ex_cldr Core Team

i would challenge the “unicode is not usually needed” assumption. Unless you have complete confidence in your knowledge of the cultural context of your audience, Latin-1 is a poor assumption. And in an Elixir context, the String.t type is unicode so its not surprising that developers are caught out by the default behaviour of Regex being Latin-1 (even though I understand why the choice, I think its hard to justify as the default).

Latin-1 doesn’t even include the Euro symbol. @michalmuskala’s native language can’t be represented in Latin-1 (Polish is Latin-2). And given the global participation of this forum I’d warrant there are many members building applications for a global audience and therefore should not make limited assumptions on character input or output.

I suppose I may be an outlier on this (Australian, living in Singapore, and having worked in several Asian countries for quite a while). It even bugs me that we don’t consistently address José with the name his mother gave him :slight_smile:

Elixir is a Unicode-centric language. I think that the :re option "u" should be the default.

Nicd

Nicd

With unicode flag:

iex(1)> Regex.match?(~r/\p{Lu}/u, "Ö")
true
iex(2)> Regex.match?(~r/\p{Lu}/u, "ö")
false

Last Post!

kip

kip

ex_cldr Core Team

Thats a great contribution, thanks!

Would you think it appropriate to also mention that it supports Unicode category notation? Like

iex> Regex.match? ~r/\p{Lu}\p{Ll}+/, "José"
true

And Unicode Block notation too:

iex> Regex.match? ~r/\p{Latin}+/, "José"
true
iex> Regex.match? ~r/\p{Hiragana}+/, "José"
false
iex> Regex.match? ~r/\p{Cherokee}+/, "José"
false

Where Next?

Trending in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
tj0
I’ve been following the steps here for the upgrade from 1.6 to 1.7 and it has gone relatively smoothly all the way till the phoenix_view ...
New
cgraham
Hi! What is currently the best library/method for parsing text and tabular data out of PDF files in Elixir or Erlang?
New
stefanchrobot
Hi, I need a way to handle data migrations in my application. I found an article by @wojtekmach about manual migrations: Automatic and ma...
New
stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
webofbits
Squid Mesh is an open source workflow automation runtime for Elixir applications. It is aimed at Phoenix and OTP apps that want to defin...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
kip
In 2021 I started a new library called Tempo with the objective of modelling time as a set of intervals - not as instants. In 2022 I gave...
New

We're in Beta

About us Mission Statement