Just tried Unicode.Regex.compile("[[:Emoji:]]") in my console and the regex correctly matched all the latest emojis. Thank you for your excellent work!
sheharyarn
0
kip
ex_cldr Core Team
Glad it does what you need it to do! And appreciate the feedback, its great for motivation 
Next version of Unicode, 14.0, will be out in September and I will have my Unicode libraries up-to-date at launch time.
1
kip
ex_cldr Core Team
Introducing the very first version of Unicode Transform which implements the CLDR Transform specification.
In this first version it implements only the Latin to ASCII transform. This is commonly thought of as “remove accents” so although its only one very small step, its possible this transform has some use to the community.
Examples
iex> Unicode.Transform.LatinAscii.transform "Considérant que la reconnaissance de la dignité inhérente à tous les membres"
"Considerant que la reconnaissance de la dignite inherente a tous les membres"
iex> Unicode.Transform.LatinAscii.transform "Da die Anerkennung der angeborenen Würde und der gleichen und unveräußerlichen Rechte aller Mitglieder"
"Da die Anerkennung der angeborenen Wurde und der gleichen und unverausserlichen Rechte aller Mitglieder"
Text which is not in the latin script (technically not in the set [[:Latin:][:Common:][:Inherited:][〇]]) is passed through unchanged.
Background
This is a fun project like most of the Unicode projects - if you can believe it! With a lot of rabbit holes. To implement this library required implementing
-
Introspection of Unicode character properties in ex_unicode.
Implementation
The implementation is in two parts:
- Generate an elixir module from the CLDR transform
.xmlfile. For example, thexmlfor theLatintoAsciitransform goes from:
# This handles only Latin, Common, and IDEOGRAPHIC NUMBER ZERO (Han).
#
:: [[:Latin:][:Common:][:Inherited:][〇]] ;
#
:: NFD() ;
[[:Latin:][0-9]] { [:Mn:]+ → ; # maps to nothing; remove all Mn following Latin letter/digit
:: NFC() ;
#
# Some of the following mappings (noted) are from CLDR ‹character-fallback› data.
# (Note, here "‹character-fallback›" uses U+2039/U+203A to avoid XML issues)
#
# Latin letters and IPA
#
Æ → AE ; # 00C6;LATIN CAPITAL LETTER AE (from ‹character-fallback›)
Ð → D ; # 00D0;LATIN CAPITAL LETTER ETH
Ø → O ; # 00D8;LATIN CAPITAL LETTER O WITH STROKE
Þ → TH ; # 00DE;LATIN CAPITAL LETTER THORN
...
becomes:
defmodule Unicode.Transform.LatinAscii do
use Unicode.Transform
# This file is generated. Manual changes are not recommended
# Source: Latin
# Target: ASCII
# Transform direction: both
# Transform alias: und-t-d0-ascii und-Latn-t-s0-ascii
# This handles only Latin, Common, and IDEOGRAPHIC NUMBER ZERO (Han).
#
filter("[[:Latin:][:Common:][:Inherited:][〇]]")
#
transform("NFD")
# maps to nothing; remove all Mn following Latin letter/digit
replace("[:Mn:]+", "", preceeded_by: "[[:Latin:][0-9]]")
#
transform("NFC")
#
# Some of the following mappings (noted) are from CLDR ‹character-fallback› data.
# (Note, here "‹character-fallback›" uses U+2039/U+203A to avoid XML issues)
#
# Latin letters and IPA
#
# 00C6;LATIN CAPITAL LETTER AE (from ‹character-fallback›)
replace("Æ", "AE")
# 00D0;LATIN CAPITAL LETTER ETH
replace("Ð", "D")
# 00D8;LATIN CAPITAL LETTER O WITH STROKE
replace("Ø", "O")
...
end
- Step two is the implementation of the macros (
filter/1,convert/3,transform/1) and others that generate the final code. This approach lets developers define their own transforms in an Elixir-friendly way.
Next steps
The current version implements a minimal part part of the standard. Although parsing and generating the module is largely conformant, the code generation is not yet complete. Therefore version 0.1.0 is only useful to people who can benefit from the Unicode.Transform.LatinAscii.transform/1.
4
amnu3387
Hi kip,
I don’t have much knowledge in this. Is the objective of CLDR Transform to allow one to parse text like those accented examples into basic ASCII while allowing one to revert it lossly back into its original “encoding”?
0
kip
ex_cldr Core Team
Unicode 14 is due for release in September. As a preview, I have released ex_unicode version 1.12.0-rc.0. The key features of Unicode 14 are:
-
Add 838 characters, for a total of 144,697 characters. These additions include 5 new scripts, for a total of 159 scripts, as well as 37 new emoji characters.
-
Add support for lesser-used languages and unique written requirements worldwide, including numerous symbols additions. Funds from the Adopt-a-Character program provided support for some of these additions. The new scripts and characters include:
- Toto, used to write the Toto language in India near Bhutan
- Cypro-Minoan, an undeciphered historical script primarily used on the island of Cyprus
- Vithkuqi, an historic script used to write Albanian, and undergoing a modern revival
- Old Uyghur, an historic script used in Central Asia and elsewhere to write Turkic, Chinese, Mongolian, Tibetan, and Arabic languages
- Tangsa, a modern script used to write the Tangsa language, which is spoken in India and Myanmar
- Many Latin additions for extended IPA
- Arabic script additions used to write languages across Africa and in Iran, Pakistan, Malaysia, Indonesia, Java, and Bosnia, and to write honorifics, and additions for Quranic use
- Other character additions support languages of the Philippines, North America, India, and Mongolia
-
Popular symbol additions:
- 37 emoji characters. For complete statistics regarding all emoji as of Unicode 14.0, see Emoji Counts. For more information about emoji additions in version 14.0, including new emoji ZWJ sequences and emoji modifier sequences, see Emoji Recently Added, v14.0.
1
kip
ex_cldr Core Team
I’m very sorry for not replying to you earlier!
What you are describing - lossless bi-directional conversion - sounds like you’re referring to character set encoding. Transforms ≠ Encoding.
Transforms are about changing representations and transliteration. For example, can we represent Chinese language in a romanised way? This is not a replacement of one Chinese ideogram into a latin character. Its a phonetic transform (in this case, can be pinyin or some other method).
0
kip
ex_cldr Core Team
Today is Unicode 14 release day. All the related libraries I maintain are tested and updated against the new version. Several are released as 1.0 versions since the APIs are stable.
ex_unicode 1.12.0
Implements Unicode introspection. Helpful to identify the script, character categories and character properties. For example:
iex> Unicode.script ?خ
"arabic"
iex> Unicode.category ?ä
:Ll
iex> Unicode.category ?A
:Lu
iex> Unicode.category ?🧐
:So
iex> Unicode.properties ?A
[
:alphabetic,
:ascii_hex_digit,
:cased,
:changes_when_casefolded,
:changes_when_casemapped,
:changes_when_lowercased,
:grapheme_base,
:hex_digit,
:id_continue,
:id_start,
:uppercase,
:xid_continue,
:xid_start
]
Unicode Set 1.0
Implements Unicode Sets that allow for flexible definition of unicode characters that can then be used as guards, to generate patterns, ranges for nimble_parsec and regular expressions. Some examples:
# The character "๓" is the thai digit `1`
iex> Unicode.Set.match? ?๓, "[[:digit:]]"
true
# Set operations allow union, insersection and difference
# This example matches on digits, but not the Thai script
iex> Unicode.Set.match? ?๓, "[[:digit:]-[:thai:]]"
false
iex> Unicode.Set.to_pattern("[{👦🏻}-{👦🏿}]")
{:ok, ["👦🏻", "👦🏼", "👦🏽", "👦🏾", "👦🏿"]}
iex> Unicode.Regex.compile("\\p{Zs}")
{:ok, ~r/[\x{20}\x{A0}\x{1680}\x{2000}-\x{200A}\x{202F}\x{205F}\x{3000}]/u}
iex> Unicode.Regex.compile("[:graphic:]")
{:ok,
~r/[\x{20}-\x{7E}\x{A0}-\x{AC}\x{AE}-\x{377}\x{37A}-\x{37F}...]/u}
Unicode Guards 1.0
Leverages unicode_set to define a set of guards for common use. Examples:
defmodule Guards do
import Unicode.Guards
def upper(<< x :: utf8, _rest :: binary >>) when is_upper(x), do: :upper
def lower(<< x :: utf8, _rest :: binary >>) when is_lower(x), do: :lower
def digit(<< x :: utf8, _rest :: binary >>) when is_digit(x), do: :digit
def whitespace(<< x :: utf8, _rest :: binary >>) when is_whitespace(x), do: :whitespace
def currency(<< x :: utf8, _rest :: binary >>) when is_currency_symbol(x), do: :currency
end
Unicode String 1.0
Implements:
- The Unicode Case Folding algorithm to provide case-independent equality checking irrespective of language or script.
- The Unicode Segmentation algorithm to detect, break or split strings into grapheme clusters, works and sentences.
Examples:
iex> Unicode.String.equals_ignoring_case? "ABC", "abc"
true
iex> Unicode.String.equals_ignoring_case? "beißen", "beissen"
true
iex> Unicode.String.split "This is a sentence. And another.", break: :word
["This", " ", "is", " ", "a", " ", "sentence", ".", " ", "And", " ", "another", "."]
iex> Unicode.String.split "This is a sentence. And another.", break: :word, trim: true
["This", "is", "a", "sentence", ".", "And", "another", "."]
iex> Unicode.String.split "This is a sentence. And another.", break: :sentence
["This is a sentence. ", "And another."]
3
kip
ex_cldr Core Team
Way back in 2016, @Qqwy launched the Unicode package on hex. It was the original inspiration for ex_unicode - which is a project that I started in 2019 because I needed to support Unicode Sets and Level 1 of Unicode Regular Expressions in order to work on CLDR Transforms. Everything in Unicode and CLDR ends up being a loooooong journey down many unexpected but very rewarding paths.
As @wojtekmach once said to me “CLDR is … vast”. Only later on did I truly understand just how vast. I still haven’t finished CLDR transforms.
Anyway, @Qqwy and I are combining efforts with the following changes:
ex_unicodewill, from the next release, be published asunicode, replacing the currently published package. Since Qqwy’s original work was the inspiration for mine the APIs are consistent and upgrades will be easy.- Qqwy becomes a co-owner of the elixir-unicode GitGub organisation.
6
kip
ex_cldr Core Team
This week Unicode 15 was announced.
Today I’ve published unicode 1.15.0 that is based upon Unicode 15. There are no required changes to unicode_set, unicode_string, unicode_guards or unicode_transform.
2
kip
ex_cldr Core Team
Up next is unicode_unihan which is a new library that introspects the Unihan database.
If you thought that Latin-1 with a 159 code points is tricky enough with majuscule and minuscule and diacritics - wait til you see Unihan.
Unihan is 98,000 code points (and counting) it’s built upon shared history, culture and politics amongst China, Japan, Korea and Vietnam. Han refers to the Chinese people; Unihan refers to the monumental task undertaken to try to produce a single set of code points that respects these historical, cultural and political contexts.
The resulting Unihan Database encapsulates a wealth of information. Which can now be introspected with a new unicode_unihan library.
Motivated and inspired by @jkwchui and the work he is doing at https://visual-fonts.com this library is a collaborative work in progress. Its early days but for anyone interested you’re very welcome to provide suggestions and feedback. Here’s a simple example:
iex> Unicode.Unihan.unihan("人")
%{
kUnihanCore2020: "GHJKMPT",
kCNS1992: "1-4429",
kFourCornerCode: ["8000.0"],
kIRGHanyuDaZidian: ["10101.100"],
kFrequency: "1",
kGradeLevel: "1",
codepoint: 20154,
kCowles: ["5115.5", "5117"],
kCNS1986: "1-4429",
kHanyuPinlu: ["rén(16866)", "ren(280)"],
kBigFive: "A448",
kHanyuPinyin: ["10101.100:rén"],
kTotalStrokes: %{"zh-Hans": 2, "zh-Hant": 2},
kIICore: ["AGTJHKMP"],
kSBGY: ["102.23"],
kCihaiT: ["80.201"],
kKPS0: ["FCC5"],
kTGHZ2013: ["313.110:rén"],
kIRGDaiKanwaZiten: ["00344"],
kTaiwanTelegraph: ["0086"],
kMorohashi: ["00344"],
kDaeJaweon: "0190.010",
kTGH: ["2013:10"],
kKSC0: ["7649"],
kMainlandTelegraph: ["0086"],
kTang: ["*njin", "njin"],
kHangul: [%{grapheme: "인", source: "0E"}],
kIRG_TSource: "T1-4429",
kJapaneseKun: ["HITO"],
kIRGKangXi: ["0091.010"],
kIRG_JSource: "J0-3F4D",
kHanYu: ["10101.100"],
kXerox: ["241:051"],
kRSAdobe_Japan1_6: ["C+2579+9.2.0"],
kIRG_KSource: "K0-6C51",
kCangjie: "O",
kFenn: ["429A"],
kCantonese: ["jan4"],
kVietnamese: ["nhân"],
kLau: ["3328"],
kGB1: "4043",
kIRG_KPSource: "KP0-FCC5",
kKoreanEducationHanja: ["2007"],
kKorean: ["IN"],
kJapaneseOn: ["JIN", "NIN"],
kRSUnicode: ["9.0"],
kGB0: "4043",
kFennIndex: ["226.01"],
kXHC1983: ["0959.010:rén"],
kMatthews: [...],
...
}
3