Bridging locale name differences between ex_cldr & gettext

@lenards, thanks for the detailed message. This is a poor error message. It’s correct as far as it goes, but it doesn’t explain well enough.

TLDR; While en-US isn’t known to CLDR, setting your CLDR to en-US will correctly link it to the Gettext locale en_US.

During the configuration phase (ie at compilation time), ex_cldr tries to resolve the required locale names by combining the requested ex_cldr locale names with the configured Gettext locale names (as you correctly identified).

At configuration time the key thing of interest is “Does CLDR have a data repository for this locale”. For en_US and de_DE it does not since by the rules of CLDR, the data in en is the data for en-US and the data for de is the data for de-DE.

At runtime, however, when validating a locale derived from the Accept-Language header or elsewhere, ex_cldr will still try to find a Gettext locale to match with the requested locale. Using your example configuration you could try this:

iex> {:ok, locale} = MyPhoenixApp.Cldr.validate_locale("en-US")
{:ok, #Cldr.LanguageTag<en-US [validated]>}
iex> locale.gettext_locale_name
"en_US"

If for some reason thats not what you see (I tested it in my dev environment and it appears to be working as expected) please do open an issue.

Thanks Kip!

All that information I included and I forgot to mention that I was seeing #Cldr.LanguageTag<de-DE [validated]> when I was sending in a header value Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7 when using the Plug.

I’ll try changing the CLDR locales quick.

Yes, thats exactly what you would expect to see!

Requesting a locale de-DE (which has a 1.0 weighting in the header and therefore the highest priority to match) will match the ex_cldr configured locale of de. It will use the extra provided information of the territory DE to note that thats the users preferred territory for localisation.

In short, configuration is about “what data do I have available in CLDR to use for localisation”. Runtime is about “what’s the best fit data available to match with what the user requested”

One way to explore what is being resolved is to look at some of the data in the language tag struct:

iex> {:ok, locale} = TestBackend.Cldr.validate_locale "en-US"
{:ok, #Cldr.LanguageTag<en-US [validated]>}

# What we asked for
iex> locale.requested_locale_name 
"en-US"

# The CLDR data repository to use
iex> locale.cldr_locale_name     
:en

# What Gettext locale we linked to
iex> locale.gettext_locale_name
"en_US"

# What territory is associated
iex> locale.territory          
:US

I might have misunderstood about how to eliminate the warning.

Leaving the MyPhoenixApp.Gettext configured the same.

If I change the CLDR configuration to:

config :ex_cldr,
  default_locale: "en-US",
  default_backend: MyPhoenixApp.Cldr,
  json_library: Jason

And then alter lib/myphxapp/cldr.ex to the following:

defmodule MyPhoenixApp.Cldr do
  use Cldr,
    locales: ["en-US", "de-DE"],
    default_locale: "en-US",
    add_fallback_locales: false,
    gettext: MyPhoenixApp.Gettext,
    data_dir: "./priv/cldr",
    otp_app: :myphxapp,
    precompile_number_formats: ["¤¤#,##0.##"],
    precompile_transliterations: [{:latn, :arab}, {:thai, :latn}],
    # we could include providers for Numbers, etc. For now: empty
    providers: [],
    generate_docs: true,
    force_locale_download: false
end

I’m failing a compile time check:

Compiling 263 files (.ex)
warning: The locales ["de-DE", "en-US"] are configured in the Elixir.MyPhoenixApp.Gettext gettext backend but are unknown to CLDR. They will be ignored by CLDR.


== Compilation error in file lib/myphxapp/cldr.ex ==
** (Cldr.UnknownLocaleError) Failed to install the locale named "en-US". The locale name is not known.
    (ex_cldr 2.31.0) lib/cldr/install.ex:84: Cldr.Install.do_install_locale_name/3
    (elixir 1.13.3) lib/enum.ex:937: Enum."-each/2-lists^foreach/1-0-"/2
    (ex_cldr 2.31.0) lib/cldr/install.ex:28: Cldr.Install.install_known_locale_names/1
    (ex_cldr 2.31.0) lib/cldr.ex:84: Cldr.install_locales/1
    (ex_cldr 2.31.0) expanding macro: Cldr.Backend.Compiler.__before_compile__/1
    lib/myphxapp/cldr.ex:1: MyPhoenixApp.Cldr (module)

So I’m still not sure I see how to tackle the warning here.

Exploring through iex is a great reminder. I definitely need to lean more on that as an approach

There isn’t a way to eliminate the warning in this case. During compilation I try to be as explicit as possible - locale matching is complicated enough as is. But if you’re running with “warnings as errors” this would be problematic so sounds like this needs to be revisited since it is just a notification - there is nothing “wrong”.

Open to suggestions - what would you suggest is the correct behaviour? Emit the message, but not as a warning? Something else?

The locales configuration must be an exact match to the locales available in CLDR. There’s no clever matching in this part.

I was seeking to understand the warning, so I’m not sure I have a great suggestion in mind right now.

The concern for me was the phrase “are unknown to CLDR” in the warning:

warning: The locales ["de-DE", "en-US"] are configured in the Elixir.MyPhoenixApp.Gettext gettext 
backend but are unknown to CLDR. They will be ignored by CLDR.

Because the locales appears in the - format, which is the convention for ex_cldr, and not for gettext I was even more curious what was up (because that seemed to imply some “smart, or savvy, evaluation” of the configuration by CLDR).

Let me ponder a bit and see there is a more helpful or “wiser” response inside me that isn’t available at the moment.

Thank you again for all the effort on the ex_cldr* libraries and your prompt, thorough responses!

Thanks for the encouragement!

I have changed both the error message and the way its printed. It no longer uses IO.warn/2 so it won’t trigger compilation errors if you use warnings_as_errors: true. The message now reads:

note: The locale "en_US" is configured in the MyApp.Gettext gettext backend but is unknown to CLDR. It will not be used to configure CLDR but it will still be used to match CLDR locales to Gettext locales at runtime.

BTW, you can also add suppress_warnings: true to your CLDR backend configuration and it won’t print warnings. I’ve just seen that its not honoured in all cases - but will be in this next release.

I’ll push a release in a few hours with this change.