kip
Unicode libraries - Fun with Unicode (introspection, lookup, sets, guards, transforms...)
Following on from my CLDR lbraries I started work on Unicode transforms. But like everything related to CLDR there is a lot of yak-shaving and rabbit-hole travelling required.
The net result is a bunch of new libraries designed to make it easier to work with Unicode blocks, scripts, categories, properties and sets. These are:
- ex_unicode that introspects a string or code point and tells you a lot more than you probably want to know. Buts is a good building block for other libraries.
- unicode_set supports the Unicode Set syntax and provides the macro
Unicode.Set.match?/2that can be used to build clever guards to match on Unicode blocks, scripts, categories and properties. - unicode_guards uses
ex_unicodeandunicode_setto provide a set of prepackaged unicode-friendly guards. Such asis_upper/1,is_lower/1,is_currency_symbol/1,is_whitespace/1andis_digit/1. - unicode_transform is a work in progress to implement the unicode transform specification and to generate transformation modules.
- unicode_string will be the last part of this series that will provide functions to split and replace strings based upon unicode sets. Work hasn’t yet started but its going to be a fun project.
Unicode sets in particular allow some cool expressions. For example:
require Unicode.Set
# Is a given code point a digit? This is the
# digit `1` in the Thai script
iex> Unicode.Set.match?(?๓, "[[:digit:]]")
true
# What if we want to match on digits, but not Thai digits?
# Use set difference!
iex> Unicode.Set.match?(?๓, "[[:digit:]-[:thai:]]")
false
Since Unicode.Set.match?/2 is a macro, all the work of parsing, extracting code points, doing set operations and generating the guard code is done at compile time. The resulting code runs about 3 to 8 times faster than a regex case. (although of course regex has a much larger problem domain).
First Post!
tmbb
You should think of renaming your modules so that Unicode.Set becomes UnicodeSet instead, or at least Unicode.UnicodeSet if you want to make it clear that everything is under the Unicode namespace. The original name (Unicode.Set) doesn’t play well with aliasing.
Most Liked
kip
Released today is Unicode Set version 0.11.0 which is primarily a bug fix release . The API, test coverage and overall stability is much improved. A version 1.0 can be expected before end of the year.
Two functional improvements may be useful:
Unicode sets for blank, graphic and print
From time-to-time on the forum there is the question “how can I detect if a string or character is printable”. In Unicode this is not a simple matter but Unicode Regular Expressions provide a portable definition of three unicode sets that may prove useful:
# `\p{blank}` is the set of "horizontal space characters"
# and is defined as `\p{gc=Space_Separator}\N{CHARACTER TABULATION}`
iex> Unicode.Set.match? "K", "[:blank:]"
false
iex> Unicode.Set.match? " ", "[:blank:]"
true
# Non breaking space
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:blank:]"
true
# Graph is that set of characters that create an impression
# and is defined as `[^\p{space}\p{gc=Control}\p{gc=Surrogate}\p{gc=Unassigned}]`
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:graph:]"
false
iex> Unicode.Set.match? " ", "[:graph:]"
false
iex> Unicode.Set.match? "克", "[:graph:]"
true
# Print is the combination of graphic and space sets minus control characters
# and is defined as `\p{graph}\p{blank}-\p{cntrl}`
iex> Unicode.Set.match? "克", "[:print:]"
true
iex> Unicode.Set.match? << 0xa0 :: utf8 >>, "[:print:]"
true
Unicode Regular Expressions
Unicode.Regex.compile/2 is now largely compliant with the Unicode Regular Expression standard. It operates by expanding unicode sets before compiling in the usual manner with Regex.compile/2.
kip
The Unicode consortium today introduced Unicode version 13.0 that adds 5,390 characters, for a total of 143,859 characters. These additions include four new scripts, for a total of 154 scripts, as well as 55 new emoji characters. As a result there are some updates to ex_unicode and related packages.
-
ex_unicode version 1.4.0 adds support for Unicode 13. It also add some additional derived categories for detecting quote marks of varying kinds (left, right, double, single, ambidextrous, all). Changelog
-
unicode_set version 0.5.0 adds support for quote-related unicode sets such as
[[:quote_mark:]],[[:quote_mark_left:]],[[:quote_mark_double:]]and so on. Changelog -
unicode_guards version 0.2.0 adds guards for quote marks. Changelog
is_quote_mark/1is_quote_mark_left/1is_quote_mark_right/1is_quote_mark_ambidextrous/1is_quote_mark_single/1is_quote_mark_double/1
Have fun with Unicode!
kip
Way back in 2016, @Qqwy launched the Unicode package on hex. It was the original inspiration for ex_unicode - which is a project that I started in 2019 because I needed to support Unicode Sets and Level 1 of Unicode Regular Expressions in order to work on CLDR Transforms. Everything in Unicode and CLDR ends up being a loooooong journey down many unexpected but very rewarding paths.
As @wojtekmach once said to me “CLDR is … vast”. Only later on did I truly understand just how vast. I still haven’t finished CLDR transforms.
Anyway, @Qqwy and I are combining efforts with the following changes:
ex_unicodewill, from the next release, be published asunicode, replacing the currently published package. Since Qqwy’s original work was the inspiration for mine the APIs are consistent and upgrades will be easy.- Qqwy becomes a co-owner of the elixir-unicode GitGub organisation.
Last Post!
kip
Lots of solid updates to several the Unicode libraries today. Overall they compile faster, run faster and are more conformant to their respective standards.
unicode 2.0.0
A major release of the base library.
-
~10x faster lookups, an order of magnitude faster compilation: Category/script/property lookups now use binary search over compact range tables instead of huge generated guard clauses.
-
unicode_guardslibrary is folded in. The separateunicode_guardspackage is no longer needed and will be retired, The guards (is_upper/1,is_lower/1,is_digit/1,is_whitespace/1, the quotation-mark guards and more) now ship withunicode. -
Correct derived categories:
:Assigned,:Graph,:Visibleand:Printableare now computed from the current character database rather than stale static tables.:Printablenow matchesString.printable?/1(it previously excluded most of the BMP, including Arabic, CJK and Hangul), and:Assignedpicked up ~16k codepoints. -
Unicode.CharacterName.to_codepoint/1resolves a character name to its codepoint (loose matching), backed by a compact sorted blob. -
Fixed UTF‑16/UTF‑32 validation in
Unicode.replace_invalid/3, which previously crashed on any input.
unicode_set 1.7.0
A large correctness pass on UnicodeSet parsing and regex generation, closing many gaps against ICU/TR35. Tested against both standards suggest the implementation is now properly conforming.
Highlights:
- the full set of escapes (
\a–\v,\xH,\u{…}/\x{…}including astral, octal,\cX), single-quote quoting, \N{name}resolution (viaunicode2.0),- correct handling of string members and string ranges when compiled to a regex,
- negated sets containing string members,
- De Morgan reduction for unions of complements (
[[^a][^b]]), - the
Is/Inproperty prefixes, - and digit-bearing block names, and left-to-right set-operation precedence.
unicode_transform 1.1.0
The pure-Elixir CLDR transform engine now conforms to 99.99% of the official CLDR transform test data, up from ~81% — roughly twenty root-cause fixes across the parser, compiler, engine and resolver. The remaining gaps a real spec ambiguity issues in a few tests in 4 of the 290 transforms.
It also ships a strict CLDR conformance suite driven by ~297k vendored test cases — one test per transform — so the engine’s conformance is measured and guarded on every run.
Trending in Announcing
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex










