kip

kip OP

ex_cldr Core Team

I’m pleased to report the first hex release of unicode_unihan. I’m not a domain expert, but thankfully @jkwchui is and the work and credit is all his. Thanks very much for a valuable contribution to the community.

I think this may be the most comprehensive introspection library of the Unicode Unihan database in any computer language.

You can try it out in Livebook. Just note that the Unihan database is huge and compile times for the library are correspondingly slow since all the parsing and decoding is done at compile time in order to present a very snappy user experience.

Run in Livebook

0
Post #21
jkwchui

jkwchui

Thank you @kip for doing most of the intelligent heavy-lifting, and being such a gracious teacher. I have learnt much through this. The code is feature-complete, and the hefty documentation should be fleshed out over the next weeks.

Besides shifting the burden to compile-time and hence resulting in snappy runtime performance, the Elixir Unihan library have a pair of interesting features that is more than the sum of its parts:

  1. decoding details into Elixir structs. For example, Unihan database may supply a binary of 1187.061 for a particular field, expecting the user to decode this based on the specs (then repeat for the other 90+ fields!) In Unicode.Unihan we decode every field for you as a Map with suitable keys and types:
%{
  page: 1187,
  position: 6,
  virtual: true
}
  1. availability of filter/1 and reject/1. These are similar to Enum.filter/1 and Enum.reject/1, taking a function as the argument to return a subset of Unihan.

The combination of sub-fields and ability to act upon them means you can search through and chain the outcomes into pipelines:

filter(
  &(&1[:kGradeLevel] <= 2 and
    &1[:kCantonese][:tone] == "1")
)
|> Enum.sort_by(
  fn {_codepoint, map} ->
    map[:kTotalStrokes][:Hant]
  end,
  :asc
)

This is very flexible and, AFAIK, unique access to this trove of knowledge compiled over two decades. The included LiveBook should offer a easy way to give this a spin with low commitment.

2
Post #22
Eiji

Eiji

How about this instead?

&get_in(&1, [Access.elem(1), :kTotalStrokes, :Hant])

Also :asc is the default, so you can leave it.

0
Post #23
jkwchui

jkwchui

Thanks for the suggestion. I’m not sure that would work. The input is a single tuple, so &2 doesn’t apply (only one term).

We can rewrite this to &get_in(elem(&1,1), [:kTotalStrokes, :Hant]), or pipe it as

Enum.sort_by(&(
    &1
    |> elem(1) 
    |> get_in([:kTotalStrokes, :Hant])
  ))

but I find the original more readable.

(About the :asc — I was toggling to tinker the first-sight output of a new user. I might be the one who got the most fun out of this :stuck_out_tongue: )

1
Post #24
Eiji

Eiji

yes, my bad - I have updated my reply

1
Post #25
kip

kip OP

ex_cldr Core Team

Following on from the spirited conversation about string capitalisation I was motivated to finish up my work on the Unicode Case Mapping Algorithm. A release candidate is now published as part of unicode_string version 1.3.0-rc.0.

Difference from the Elixir String casing functions

The implementation is locale aware and implements all of the transforms in SpecialCasing.txt including the conditional mappings for Greek, Turkish, Azeri and Lithuanian (Elixir’s string functions do not handle Lithuanian since they are conditional transforms).

  • Supports uppercasing Greek (not specified in Unicode). Feedback is requested on the implementation, see the last section of the post on this topic.

  • Supports titlecasing the “IJ” dipthong in Dutch (also not specified in Unicode).

  • Titlecasing works on individual words which is different to String.capitalize/2.

  • Planned support for other languages as I learn more about common practise. Feedback and feature requests are welcome.

Examples

# Basic case transformation
iex> Unicode.String.upcase("the quick brown fox")
"THE QUICK BROWN FOX"

# Dotted-I in Turkish and Azeri
iex> Unicode.String.upcase("Diyarbakır", locale: :tr)
"DİYARBAKIR"

# Upper case in Greek removes diacritics
iex> Unicode.String.upcase("Πατάτα, Αέρας, Μυστήριο", locale: :el)
"ΠΑΤΑΤΑ, ΑΕΡΑΣ, ΜΥΣΤΗΡΙΟ"

# Lower case Greek with a final sigma
iex> Unicode.String.downcase("ὈΔΥΣΣΕΎΣ", locale: :el)
"ὀδυσσεύς"

# Title case Dutch with leading dipthong
iex> Unicode.String.titlecase("ijsselmeer", locale: :nl)
"IJsselmeer"

Seeking feedback on Greek uppercasing

There is conflicting information on how to uppercase Greek. I would greatly welcome any feedback from native Greek speakers on what is correct in common usage. Here is what I have consolidated (and is extracted from the docs):

CLDR algorithm (current implementation)

According to CLDR all accents on all characters are are omitted when upcasing. This is based upon the CLDR el-Upper text transform:

  Remove 0301 following Greek, with possible intervening 0308 marks.
  ::NFD();
  For uppercasing (not titlecasing!) remove all greek accents from greek letters.
  This is done in two groups, to account for canonical ordering.
  [:Greek:] [^[:ccc=Not_Reordered:][:ccc=Above:]]*? { [\u0313\u0314\u0301\u0300\u0306\u0342\u0308\u0304] → ;
  [:Greek:] [^[:ccc=Not_Reordered:][:ccc=Iota_Subscript:]]*? { \u0345 → ;
  ::NFC();

That transform basically says remove all accents except a subscripted iota. It doesn’t handle dipthongs correctly.

Mozilla algorithm

Mozilla has a thread on a bug report
that:

Greek accented letters should be converted to the respective non-accented uppercase
letters. The required conversions are the following (in Unicode):

ά → Α
έ → Ε
ή → Η
ί → Ι
ΐ → Ϊ
ό → Ο
ύ → Υ
ΰ → Ϋ
ώ → Ω

Also diphthongs (two-vowel constructs) should be converted as follows, when the
first vowel is accented:

άι → ΑΪ
έι → ΕΪ
όι → ΟΪ
ύι → ΥΪ
άυ → ΑΫ
έυ → ΕΫ
ήυ → ΗΫ
όυ → ΟΫ

That thread seems to align with current-day Mozilla which says the rules are:

In Greek (el), vowels lose their accent when the whole word is in
uppercase (ά/Α), except for the disjunctive eta (ή/Ή). Also, diphthongs
with an accent on the first vowel lose the accent and gain a diaeresis
on the second vowel (άι/ΑΪ).

3
Post #26
bottlenecked

bottlenecked

I think I’m going to agree with the mozilla bug report (caveat: not a linguist). In common usage there’s two diacritic marks that survive:

  • the accent (as in ά or Ά - applicable to all vowels)
  • the dialytics (as in ϋ or Ϋ - applicable to some vowels only - ι and υ) - also mentioned as diaeresis in the text quoted above

The accent is used to denote where to stress the pronunciation. When capitalizing (e.g from άλλος -> Άλλος) the accent survives if it’s on the first letter of the word, but not when upppercasing (e.g. from άλλος -> ΑΛΛΟΣ)

The dialytics are also used to inform pronunciation: for example αυλή (means yard / backyard, pronounced ‘avlee’) does not require the dialytics because αυ is meant to be read as a diphthong, like ‘av’.

However in the word άυλη (means without matter / ethereal) notice that the accent is now on ά - which changes the pronunciation to something like ‘aelee’. When uppercasing we would do it as ΑΫΛΗ - so dialytics would be inserted to preserve the pronunciation since now there is no accent on Α to disambiguate (but if we were capitalizing, it would become Άυλη - the accent mark would be preserved because it’s on the first letter of the word and dialytics don’t have to be inserted for disambiguation)

There’s also the case where both an accent and dialytics are present on the same vowel - but the same rules apply as above for dialytics: the accent goes away when uppercasing, the dialytics either stay or get added as needed.

I hope that helps some

In the word ὀδυσσεύς you posted above, the has a diacritic mark called ‘psili’ but that is no longer in use so for this and other marks you probably shouldn’t spend any time on

1
Post #27
kip

kip OP

ex_cldr Core Team

I’ve just published unicode_string version 1.4.0. The primary update is the addition of dictionary-based word breaking for certain locales.

Not all languages use whitespace to separate words (most commonly East Asian languages) so a dictionary lookup is more appropriate - although not perfect.

The motivation to get this done now it to support the release in 2 weeks of ex_cldr_person_names.

Supported locales for dictionary-based word breaking

This implementation supports dictionary-based word breaking for:

  • Chinese (zh, zh-Hant, zh-Hans, zh-Hant-HK, yue, yue-Hans) locales,
  • Japanese (ja) using the same dictionary as for Chinese,
  • Thai (th),
  • Lao (lo),
  • Khmer (km) and
  • Burmese (my).

The dictionaries implemented are those used in CLDR since they are under an open source license and also for consistency with ICU.

Downloading dictionaries

Note that these dictionaries need to be downloaded with mix unicode.string.download.dictionaries prior to use. Each dictionary will be parsed and loaded into persistent_term on demand.

Memory usage

Note that each dictionary has a sizable memory footprint as measured by :persistent_term.info/0:

Dictionary Memory Mb
Chinese 104.8
Thai 9.6
Lao 11.4
Khmer 38.8
Burmese 23.1

Examples

iex>  Unicode.String.split("明德", locale: :en) == 
["明", "德"]

iex> Unicode.String.split("明德", locale: :zh_Hant_HK) == 
["明德"]

iex> Unicode.String.split("สวัสดีเจ้านาย", locale: :th) 
["สวัสดี", "เจ้า", "นาย"]

iex> Unicode.String.split("ສະບາຍດີນາຍຈ້າງ", locale: :lo)
["ສະບາຍດີ", "ນາຍ", "ຈ້າງ"]

iex> Unicode.String.split("ສະမင်္ဂလာပါ သူဌေး", locale: :my)
["ສ", "ະ", "မင်္ဂလာ", "ပါ", " ", "သူဌေး"]

iex> Unicode.String.split("ສជំរាបសួរចៅហ្វាយ", locale: :km)
["ສ", "ជំរាបសួរ", "ចៅហ្វាយ"]
4
Post #28
kip

kip OP

ex_cldr Core Team

Unicode 16 was introduced today and with it an update to the Elixir library Unicode version 1.20.0.

Summary of changes to Unicode 16

Unicode 16.0 adds 5,185 characters, for a total of 154,998 characters. The new additions include seven new scripts:

  • Garay is a modern-use script from West Africa.
  • Gurung Khema, Kirat Rai, Ol Onal and Sunuwar are four modern-use scripts from Northeast India and Nepal.
  • Todhri is an historic script used for Albanian.
  • Tulu-Tigalari is an historic script from Southwest India.

Other character additions include seven new emoji characters plus 3,995 additional Egyptian Hieroglyphs and over 700 symbols from legacy computing environments.

In addition to new characters, new “Moji Jōhō Kiban” (文字情報基盤) Japanese source references have been added for over 36,000 CJK unified ideographs. This additional data will be reflected in an upcoming release of unicode_unihan.

Do not emit data

This is a new data set that collects information about characters or character sequences that should not be emitted or generated in newly authored text and for which a suitable alternative sequence exists. This data could be used by applications such as input methods or autocorrect. This data will be incorporated into an upcoming update to unicode.

1
Post #29
kip

kip OP

ex_cldr Core Team

I’ve updated unicode_set and unicode_string to be compatible with OTP 28.

The only material change is to convert all compile-time regex’s to be runtime. In unicode_string this may have some performance implications since the entire string break process is an iterative application of rules that are regex’s.

0
Post #30