Released today is Unicode Set version 0.11.0 which is primarily a bug fix release . The API, test coverage and overall stability is much improved. A version 1.0 can be expected before end of the year.

Two functional improvements may be useful:

Unicode sets for blank, graphic and print

From time-to-time on the forum there is the question “how can I detect if a string or character is printable”. In Unicode this is not a simple matter but Unicode Regular Expressions provide a portable definition of three unicode sets that may prove useful:

# `\p{blank}` is the set of "horizontal space characters" 
# and is defined as `\p{gc=Space_Separator}\N{CHARACTER TABULATION}`
iex> Unicode.Set.match? "K", "[:blank:]"
false
iex> Unicode.Set.match? " ", "[:blank:]"
true
# Non breaking space
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:blank:]"
true

# Graph is that set of characters that create an impression 
# and is defined as `[^\p{space}\p{gc=Control}\p{gc=Surrogate}\p{gc=Unassigned}]`
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:graph:]"
false
iex> Unicode.Set.match? " ", "[:graph:]"
false
iex> Unicode.Set.match? "克", "[:graph:]"              
true

# Print is the combination of graphic and space sets minus control characters
# and is defined as `\p{graph}\p{blank}-\p{cntrl}`
iex> Unicode.Set.match? "克", "[:print:]"
true
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:print:]"
true

Unicode Regular Expressions

Unicode.Regex.compile/2 is now largely compliant with the Unicode Regular Expression standard. It operates by expanding unicode sets before compiling in the usual manner with Regex.compile/2.

The Unicode consortium today introduced Unicode version 13.0 that adds 5,390 characters, for a total of 143,859 characters. These additions include four new scripts, for a total of 154 scripts, as well as 55 new emoji characters. As a result there are some updates to ex_unicode and related packages.

• ex_unicode version 1.4.0 adds support for Unicode 13. It also add some additional derived categories for detecting quote marks of varying kinds (left, right, double, single, ambidextrous, all). Changelog

• unicode_set version 0.5.0 adds support for quote-related unicode sets such as [[:quote_mark:]], [[:quote_mark_left:]], [[:quote_mark_double:]] and so on. Changelog

• unicode_guards version 0.2.0 adds guards for quote marks. Changelog

  • is_quote_mark/1
  • is_quote_mark_left/1
  • is_quote_mark_right/1
  • is_quote_mark_ambidextrous/1
  • is_quote_mark_single/1
  • is_quote_mark_double/1

Have fun with Unicode!

Way back in 2016, @Qqwy launched the Unicode package on hex. It was the original inspiration for ex_unicode - which is a project that I started in 2019 because I needed to support Unicode Sets and Level 1 of Unicode Regular Expressions in order to work on CLDR Transforms. Everything in Unicode and CLDR ends up being a loooooong journey down many unexpected but very rewarding paths.

As @wojtekmach once said to me “CLDR is … vast”. Only later on did I truly understand just how vast. I still haven’t finished CLDR transforms.

Anyway, @Qqwy and I are combining efforts with the following changes:

• ex_unicode will, from the next release, be published as unicode, replacing the currently published package. Since Qqwy’s original work was the inspiration for mine the APIs are consistent and upgrades will be easy.
• Qqwy becomes a co-owner of the elixir-unicode GitGub organisation.