lenards

Bridging locale name differences between ex_cldr & gettext

I did search on the forums here to see if there was a previous answer and didn’t come up with a clear match.

I’m working on a Phoenix app that is using gettext for text translations. I user will have a preferred language, so we’ll be serving up pages using that. However, for unauthenticated pages, we’ll need to read and respond to the Accept-Language headers.

I’ve been evaluating using the Plug from ex_cldr_plug and it’s remarkably robust. However it seems like the locale naming for Gettext isn’t aligned with those in CLDR.

ling 893 files (.ex)
warning: The locales ["de-DE", "en-US"] are configured in the Elixir.MyPhoenixApp.Gettext gettext backend but are unknown to CLDR. They will be ignored by CLDR.

Generating MyPhoenixApp.Cldr for 3 locales named [:de, :en, :und] with a default locale named :en

config/config.ex

# Configure for Internationalization (i18n)
config :myphxapp, MyPhoenixApp.Gettext,
  default_locale: "en_US",
  locales: ["en_US", "de_DE"]

config :ex_cldr,
  default_locale: "en",
  default_backend: MyPhoenixApp.Cldr,
  json_library: Jason

lib/myphxapp/cldr.ex

defmodule MyPhoenixApp.Cldr do
    use Cldr,
        locales: ["en", "de"],
        default_locale: "en",
        add_fallback_locales: false,
        gettext: MyPhoenixApp.Gettext,
        data_dir: "./priv/cldr",
        otp_app: :myphxapp,
        precompile_number_formats: ["¤¤#,##0.##"],
        precompile_transliterations: [{:latn, :arab}, {:thai, :latn}],
        # we could include providers for Numbers, etc. For now: empty
        providers: [],
        generate_docs: true,
        force_locale_download: false
end

lib/myphxapp/gettext.ex

defmodule MyPhoenixApp.Gettext do
  use Gettext, otp_app: :myphxapp
end

I believe that Gettext is using POSIX locale name from what I can see in an earlier version of the ex_cldr readme:

Since Gettext uses the Posix locale name format (locales with an ‘_‘ in them) and Cldr uses the Unicode format (a ‘-‘ as the subtag separator), Cldr will transliterate locale names from Gettext into the Cldr canonical form.

I imagine the warning is hinting at this being more than just a format definition though (so not just underscore _ vs dash -).

I realize that de_DE is not populated in CLDR from Kip’s response here:

Locales configuration in Cldr - #4 by kip

Is there a way to address the warning and allow Gettext’s locales to coexist with CLDR’s definitions?

Or, I’m not sure if the correct move to is change from en_US & de_DE to simply en and de?

I know that the application will need to support en_GB and en_CA soon, so that’s why I was attempting so use the format for language-code / country code.

Hopefully I’ve provided enough configuration details (and not too much).

Thanks in advance

12 comments

#gettext #ex_cldr

2 1712 12

2022-07-26 18:57:17 UTC

Marked As Solved

kip

ex_cldr Core Team

I have published ex_cldr version 2.32.1 with a change to the error message and no longer using IO.warn/2. The changelog entry reads:

Bug Fixes

Don’t use IO.warn/2 when compiling a backend and a known Gettext locale can’t be matched to a Cldr locale. IO.warn/2 will cause errors if the compilation setting warnings_as_errors: true is set. Instead, these messages will be output as a “note” that does not trigger warnings. In addition the error message has been improved to make clear that although the Gettext locale has no Cldr equivalent, it will still be matched at runtime. See the conversation at https://forum.elixirforum.com/t/bridging-locale-name-differences-between-ex-cldr-gettext. Thanks to @lenards for the report.

Post #12

Also Liked

kip

ex_cldr Core Team

@lenards, thanks for the detailed message. This is a poor error message. It’s correct as far as it goes, but it doesn’t explain well enough.

TLDR; While en-US isn’t known to CLDR, setting your CLDR to en-US will correctly link it to the Gettext locale en_US.

During the configuration phase (ie at compilation time), ex_cldr tries to resolve the required locale names by combining the requested ex_cldr locale names with the configured Gettext locale names (as you correctly identified).

At configuration time the key thing of interest is “Does CLDR have a data repository for this locale”. For en_US and de_DE it does not since by the rules of CLDR, the data in en is the data for en-US and the data for de is the data for de-DE.

At runtime, however, when validating a locale derived from the Accept-Language header or elsewhere, ex_cldr will still try to find a Gettext locale to match with the requested locale. Using your example configuration you could try this:

iex> {:ok, locale} = MyPhoenixApp.Cldr.validate_locale("en-US")
{:ok, #Cldr.LanguageTag<en-US [validated]>}
iex> locale.gettext_locale_name
"en_US"

If for some reason thats not what you see (I tested it in my dev environment and it appears to be working as expected) please do open an issue.

Post #2

kip

ex_cldr Core Team

One way to explore what is being resolved is to look at some of the data in the language tag struct:

iex> {:ok, locale} = TestBackend.Cldr.validate_locale "en-US"
{:ok, #Cldr.LanguageTag<en-US [validated]>}

# What we asked for
iex> locale.requested_locale_name 
"en-US"

# The CLDR data repository to use
iex> locale.cldr_locale_name     
:en

# What Gettext locale we linked to
iex> locale.gettext_locale_name
"en_US"

# What territory is associated
iex> locale.territory          
:US

Post #5

lenards

Thanks Kip!

All that information I included and I forgot to mention that I was seeing #Cldr.LanguageTag<de-DE [validated]> when I was sending in a header value Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7 when using the Plug.

I’ll try changing the CLDR locales quick.

Post #3