Mapping a language to a font charset

@mayel thats an interesting question! I don’t believe CLDR has that data built in but I’ll check the links you posted and see what is possible. (I’m the ex_cldr author).

Thanks the approach I took temporarily is to have a dropdown like this:

Screenshot 2022-07-06 at 13.58.11588×282 53.9 KB

This controls which CSS file is included in the template which in turn loads differents fonts and different charsets, e.g. if the user selects a font with “Noto Sans (More Languages)” then noto-sans-more.css is used from https://github.com/bonfire-networks/bonfire-app/tree/main/assets/static/fonts

There’s two limitations with this I’d like to improve on:

• The user shouldn’t have to worry about manually changing the font charaset, this should instead follow their locale (so they just select a general font family)
• Currently this only covers a few charsets, but I’d like to add support for more languages - see the seperate per-language fonts available - this screenshots only shows a few of them:
  
  

  Screenshot 2022-07-06 at 14.03.33580×1992 72 KB

Having this kind of logic would also allow loading different fonts/charsets not only based on the user’s locale, but based on the language of a piece of text on the page (when dealing with user-generated content)…

You can get the script for any locale:

iex> {:ok, locale} = Cldr.validate_locale("th")
{:ok, #Cldr.LanguageTag<th [validated]>}
iex> locale.script
:Thai

Which is a starting point. I will add a function Cldr.Locale.script_from_locale/1 in the next release to make this more explicit.

Now to mapping the script to a font name may be out of scope for ex_cldr but I’m curious so will do some more investigating too.

Ah that’s a good start thanks! If out of scope for ex_cldr I’ll likely package what we end up with into a library (though it will probably be opinionated, as in supporting specific font families).

After some more digging:

• I published ex_cldr version 2.31.0 which includes a new function Cldr.Locale.script_from_locale/1 to return the script. For example :Latn or :Thai or :Deva.
• You can use the function Cldr.Validity.known(:scripts) to get the name of all the 167 scripts known to CLDR. This function parses data so its only used at compile-time in ex_cldr to build functions. Its not public API so use at your own risk (its not going away however). Cldr.Locale.language_data/0 returns a lot of data that can be used to map between languages, scripts and territories its more complex data but more complete.
• Google Fonts is really sloppy in its terminology using “charset” interchangeably with “subset” when it really means “script”. And it uses “language” when it means “script” too. Disappointing really since Google is the largest sponsor of CLDR.
• CLDR does have mappings from language to script so a lookup is possible and I’ll add a function to ex_cldr to make that easy. But there’s another problem needs solving first …
• The language names used in Google Fonts are not canonical and have no alignment with ISO 639 language codes making it difficult to map from locale.script or local.language to the right font “subset”.

Hopefully this gives you something to go on. Its an interesting and useful objective to deliver a typeface to a user that can represent the requested script. So I’m happy to help on the ex_cldr - just reply here or open an issue on GitHub.

Last thought for now. Google says “language” but then uses “Latin” and “Latin-ext” which whilst a language, it isn’t used in that context. So manual mapping is going to be required, Perhaps something like this:

1. Get the ISO 639 language code from a locale

iex> {:ok, locale} = Cldr.validate_locale("th")
{:ok, #Cldr.LanguageTag<th [validated]>}
iex> locale.language
"th"

2. Manually build a map of Google Font “languages” to ISO 639 codes and then use that to match with the locale and therefore know what the Google Fonts “subset” is.

3. In building the map, you could use the data returned from Cldr.Locale.language_data/0 that tells you probably more than you want in order to help build the right mapping. For example you could map all the languages that use the latn script into the Google Fonts latin, latin-ext subset.

Yeah manual mapping is where I assumed this was headed… Thanks for all your help, glad I’m not the only one who geeks out on this stuff Will report back!

Ah this seems useful for detecting what to use for user generated text (if no language was specified by the user), though unfortunately only has the common charsets and not the additional languages: char-subsets/charlists at master · strarsis/char-subsets · GitHub

Maybe you’ll find my text library useful? It has a native Elixir implementation of text detection. You need to install also the text_corpus_udhr library to have a corpus to work with for detection:

iex> Text.detect """
Договоренность о разблокировке украинских портов для возобновления экспорта зерна остается труднодостижимой, потому что Москва использует переговоры для реализации своих военных целей и обеспечения доминирования в Черном море, заявил заместитель министра экономики и торговый представитель Украины Тарас Качка.
"""
{:ok, "ru"}