Erli18n - GNU gettext-compatible i18n for Erlang/OTP (a learning project - feedback very welcome)

Quick update for anyone following erli18n: v0.2.0 is on Hex.

This release adds named %{var} interpolation on top of the gettext API. It’s
fully additive — the existing gettext / ngettext / pgettext / npgettext
families are unchanged, so upgrading is a no-op for code you already wrote.

There’s a new f-suffix family (gettextf, ngettextf, pgettextf, npgettextf,
plus the d / dc variants). Each one resolves the translation exactly like its
non-f sibling, then splices values from a trailing bindings map by name. Binding
by name means a translator can reorder or repeat a variable and it still resolves —
wording is decoupled from argument order. It’s total and fail-soft (never raises;
an unbound %{name} is left literal), with anti-DoS output/expansion caps.

Basic — one named variable:

%% .po →  msgid "Hello, %{name}!"
%%        msgstr "Olá, %{name}!"

<<"Olá, Eduardo!"/utf8>> =
    erli18n:gettextf(my_domain, <<"Hello, %{name}!">>, <<"pt_BR">>,
                     #{name => <<"Eduardo">>}).

Advanced — pluralization with an auto-bound %{count}, plus name-based reordering:

%% .po →  msgid "%{count} tree"
%%        msgid_plural "%{count} trees"
%%        msgstr[0] "%{count} árvore"
%%        msgstr[1] "%{count} árvores"

%% ngettextf auto-binds count => N, so %{count} is always available — you never
%% pass it yourself, and the CLDR plural form for N is selected for you.
<<"1 árvore"/utf8>>   = erli18n:ngettextf(my_domain, <<"%{count} tree">>, <<"%{count} trees">>, 1,  <<"pt_BR">>, #{}).
<<"42 árvores"/utf8>> = erli18n:ngettextf(my_domain, <<"%{count} tree">>, <<"%{count} trees">>, 42, <<"pt_BR">>, #{}).

%% Because binding is by NAME, the translation can swap the order freely:
%%   msgid  "%{user} sent %{item}"        (English order: user, then item)
%%   msgstr "%{item} enviado por %{user}"  (pt_BR order: item, then user)
<<"report.pdf enviado por Ana">> =
    erli18n:gettextf(my_domain, <<"%{user} sent %{item}">>, <<"pt_BR">>,
                     #{user => <<"Ana">>, item => <<"report.pdf">>}).

Callable from Elixir too, since it’s a normal Hex dep: :erli18n.gettextf(...)

Quick update for anyone following erli18n: v0.3.0 is on Hex.

This release adds opt-in locale negotiation and a lookup-time fallback chain.
Both are additive and off by default — upgrading is a no-op, the exact-match
lookup hot path is byte-for-byte unchanged, and every existing call behaves exactly
as in 0.2.0.

Two new pieces:

1. Request-time negotiation — pick the best locale a client supports from the
   ones you actually loaded, with BCP-47 canonicalization (hyphen/underscore, casing,
   a closed legacy-alias set). parse_accept_language/1 turns an HTTP header into a
   priority-ordered list; negotiate/2 resolves it and always returns a usable
   locale (it defaults to your default_locale on no match).
2. Lookup-time fallback chain (erli18n.locale_fallback, default off) — when
   enabled, a lookup that misses the exact locale walks a canonicalization-aware
   RFC 4647 chain before returning the msgid, so a pt_BR user reads a loaded
   pt catalog instead of seeing the raw key.

Basic — negotiate the best locale from an Accept-Language header:

Available = [<<"en">>, <<"pt">>, <<"de">>],   %% the locales you actually loaded

%% Hyphenated / mixed-case / legacy tags canonicalize to match.
{ok, <<"pt">>} = erli18n:negotiate([<<"pt-BR">>], Available),

%% Straight from an HTTP header (q-values respected, q=0 dropped):
Prefs = erli18n:parse_accept_language(<<"fr-CH, de;q=0.9, en;q=0.5">>),
%%   Prefs => [{<<"fr-ch">>,1000}, {<<"de">>,900}, {<<"en">>,500}]   (q in milli-units)
{ok, <<"de">>} = erli18n:negotiate(Prefs, Available),

%% One-off tag canonicalization to the catalog-key shape:
<<"pt_BR">> = erli18n:canonicalize_locale(<<"PT-br.UTF-8">>).

Advanced — the opt-in lookup fallback chain (with canonicalization and an explicit override):

%% Only a "pt" catalog is loaded (msgid "Hello" -> "Olá"); a user asks for "pt_BR".

%% Default (off) — exact match only, identical to 0.2.0:
<<"Hello">>     = erli18n:gettext(my_domain, <<"Hello">>, <<"pt_BR">>),   %% miss -> raw msgid

%% Turn the chain on (app env, or at runtime):
ok = erli18n:set_locale_fallback(base_language),
<<"Olá"/utf8>> = erli18n:gettext(my_domain, <<"Hello">>, <<"pt_BR">>),   %% pt_BR -> pt

%% Canonicalization covers separators, case, and POSIX charset suffixes —
%% these all resolve the same loaded "pt" catalog:
<<"Olá"/utf8>> = erli18n:gettext(my_domain, <<"Hello">>, <<"pt-BR">>),
<<"Olá"/utf8>> = erli18n:gettext(my_domain, <<"Hello">>, <<"pt_BR.UTF-8">>),

%% It works across all four lookup families, so plurals fall back too:
<<"árvores"/utf8>> = erli18n:ngettext(my_domain, <<"tree">>, <<"trees">>, 2, <<"pt_BR">>),

%% Need a custom mapping? {explicit, Map} overrides specific locales
%% (unlisted ones fall through to base_language):
ok = erli18n:set_locale_fallback({explicit, #{<<"pt_BR">> => [<<"pt">>]}}),
<<"Olá"/utf8>> = erli18n:gettext(my_domain, <<"Hello">>, <<"pt_BR">>).

A typical web handler negotiates once per request, then sets the locale:

Prefs        = erli18n:parse_accept_language(AcceptLanguageHeader),
{ok, Locale} = erli18n:negotiate(Prefs, my_supported_locales()),
erli18n:setlocale(Locale).

There’s also a new opt-in [erli18n, locale, fallback] telemetry event (under the
existing emit_lookup_telemetry flag) that fires when a non-exact locale serves a
translation, with a chain_depth measurement — so you can see how often clients lean
on the fallback. The negotiation engine is pure, total, and dependency-free, and the
whole feature stays off the exact-hit path.

(Callable from Elixir too, since it’s a normal Hex dep: :erli18n.negotiate(...).)

Full notes are in the CHANGELOG. As always, feedback — especially the critical
kind — is hugely appreciated.

• Hex: https://hex.pm/packages/erli18n
• Docs: https://hexdocs.pm/erli18n
• GitHub: https://github.com/eagle-head/erli18n

ETS might be lock free access, but still does a copy per access. For rarely changing values like translations :persistent_term can remove the term copying on reads by requiring a global garbage collection on writes. Unless you’re doing heavy runtime modifications to translations that tradeoffs is probably worthwhile.

@LostKobrakai — thank you, this was exactly right, and it changed the project’s route.

You nailed the distinction I’d glossed over: “lock-free” is not “copy-free.” ETS read_concurrency removes the contention, but ets:lookup/2 still copies the matched term onto the calling process heap on every read — and for the short binaries that UI strings usually are, that copy is real on the common path. persistent_term, as you said, lets reads return a pointer into the literal area with no copy at all, paid for by a node-wide GC on writes. And your closing caveat was the heart of it: that tradeoff is worthwhile unless you’re doing heavy runtime modifications to translations.

So I did the measured-before-you-swap thing rather than rushing it: I wrote a dedicated ETS-vs-persistent_term benchmark for the read hot path. For this read-mostly, load-once workload, persistent_term reads came out ~55% faster (copy-free), which matched your reasoning, so I shipped the migration.

erli18n 0.4.0 is now live on Hex with exactly the improvement you suggested:
the catalog storage engine moved from ETS to persistent_term. One persistent term per {Domain, Locale} catalog now holds a single map of the entries plus its header; reads are copy-free and lock-free from the calling process, and a whole-catalog (re)load is a single persistent_term:put.

And I took your tradeoff on honestly rather than hiding it: reload/3,4 and unload/2 now pay the node-wide literal-area GC you described — once per (re)load. That’s negligible for the load-once-at-boot case erli18n targets and costlier under heavy runtime catalog churn, so it’s documented as a tradeoff in the CHANGELOG and the module docs. The public gettext API and all the lookup/fallback semantics are unchanged.

Full write-up in the 0.4.0 post below. Genuinely grateful — this is the kind of feedback I was hoping for.

Quick update for anyone following erli18n: v0.4.0 is on Hex.

This release is an internal storage-engine migration: the translation-catalog
substrate moved from ETS to persistent_term. It’s fully invisible to
the public API — the gettext / ngettext / pgettext / npgettext families,
the f-suffix interpolation family, and all lookup / fallback / idempotency
semantics are byte-for-byte unchanged, so upgrading is a no-op for code you
already wrote.

This one came straight from forum feedback (thank you, @LostKobrakai — more
on that below). The short version: lock-free is not copy-free. The old design
read catalog rows from a lock-free ETS table, but ets:lookup/2 still copies
the matched term onto the calling process heap on every read. For UI strings —
often short, rarely changing, loaded once — that per-read copy is real cost on
the hot path.

What changed:

• New module erli18n_pt_store — one persistent term per {Domain, Locale}
  catalog, holding a single map of the catalog’s entries plus its header. Reads
  are now persistent_term:get/2 + maps:get/3, both copy-free: the term
  lives in the literal area and lookups return a pointer into it, so even short
  binaries are no longer copied onto the caller heap. A whole-catalog (re)load is
  a single persistent_term:put/2.
• Measured, not assumed. A dedicated ETS-vs-persistent_term benchmark
  confirmed the win: persistent_term reads are ~55% faster on the read hot
  path for this read-mostly, load-once workload.
• Cleanup bonus. Because persistent_term is node-global and runtime-owned,
  a worker crash destroys nothing — so the entire ETS crash-durability machinery
  is gone: the erli18n_table_owner heir module, the ETS-TRANSFER / give_away
  handoff, and the secondary catalog index all retired. The supervisor collapses
  to a single erli18n_server child under one_for_one.

The honest tradeoff (the one @LostKobrakai named): reload/3,4 and unload/2
now trigger a node-wide persistent_term literal-area GC — paid once per
(re)load. For erli18n’s target — load catalogs once at boot — this is
negligible. Under heavy runtime catalog churn (aggressive reload) it’s a real
cost the previous ETS storage didn’t have. It’s documented up front in the
CHANGELOG and in the erli18n_server / erli18n_pt_store module docs, never
hidden.

(The project already used persistent_term for the read-mostly CLDR plural AST,
so this just extends an idiomatic pattern to the catalogs themselves.)

(Callable from Elixir too, since it’s a normal Hex dep — nothing about the call
surface changed: :erli18n.gettext(...).)

Full notes are in the CHANGELOG. As always, feedback — especially the critical
kind — is hugely appreciated.

• Hex: https://hex.pm/packages/erli18n
• Docs: https://hexdocs.pm/erli18n
• GitHub: https://github.com/eagle-head/erli18n

erli18n 0.5.0 is out on Hex, along with a new companion rebar3 plugin for the catalog workflow.

The new piece is rebar3_erli18n, a build-only (opt-in) plugin that adds four rebar3 commands:

• extract — scans source for gettext/ngettext/pgettext call sites and writes a .pot per domain
• merge — merges a .pot into a .po, keeping existing translations, fuzzy-matching renamed ids and marking removed ones obsolete
• report — translated/fuzzy/missing counts per {domain, locale}
• check — fails in CI when the committed catalogs have drifted from the source

Same limitation as gettext: only literal message ids are statically extractable. gettext(<<"Hello">>) is found; gettext(Var) still translates at runtime but won’t be picked up by the scan.

The runtime library bump is minor and additive: erli18n_po:escape_string/1 is now exported so the library and the tooling share the same PO serializer, plus some PO-parser hardening. The lookup/interpolation/fallback API is unchanged, so upgrading is a no-op.

erli18n 0.6.0 is out on Hex, along with rebar3_erli18n 0.1.1.

The new piece is optional per-request locale middleware for Cowboy and Elli. It negotiates the request locale (query string, then cookie, then the Accept-Language header — configurable) and calls erli18n:setlocale/1 before your handler runs, so handlers translate with no locale argument.

For a Phoenix/Plug stack: erli18n has no Elixir dependency. Use the Cowboy adapter, or call :erli18n.setlocale(locale) from a small Plug in the request process — the same spot you would call Gettext.put_locale/1. The canonical locale form (“pt_BR”) already matches Gettext’s, so the two agree if you set both from that one place.

cowboy and elli are optional (declared in optional_applications), so the published package still builds on kernel + stdlib alone — you add whichever you already use. The adapters share a pure core, erli18n_http, you can call directly for frameworks they don’t cover, and there is a new erli18n:loaded_locales/0 for the set of locales you have actually loaded.

Also fixed: interpolation truncation now respects UTF-8 codepoint boundaries (no split codepoints at the size cap), and erli18n_po:escape_string/1 is now total over any binary. rebar3_erli18n 0.1.1 makes extract and merge return a clean error instead of crashing when a catalog file can’t be written.

erli18n 0.7.0 is out on Hex, along with rebar3_erli18n 0.2.0.

The new piece is optional compile-time catalog codegen. The rebar3 plugin’s new compile provider (rebar3 erli18n compile) reads each (Domain, Locale) .po, parses it and compiles its Plural-Forms rule ahead of time, and emits a small generated carrier module that holds the already-parsed entries and the already-compiled rule. The app calls erli18n:register_compiled_catalogs/1 once at startup, and boot does no .po read, no parse, and no plural compile.

If you know Gettext: this is the opt-in “translations baked in at build time” path, in the same spirit as Gettext embedding your PO data at compile time — except erli18n keeps runtime .po loading (erli18n:ensure_loaded/3,4) as the default, so you choose per project. erli18n has no Elixir dependency; from a Phoenix/Plug app you would call :erli18n.register_compiled_catalogs(:my_app) from your application start/2, the same one-time boot setup you already do.

It is opt-in and additive: the provider is a no-op unless {compiled_catalogs, true} is set in rebar.config, the runtime path and the read hot path are unchanged, and register_compiled_catalogs/1 is idempotent and composes with runtime loading. Two build-time extras: an opt-in key-existence check (off | warn | strict) for facade call sites with no matching compiled key, and size/entry caps that mirror the runtime loader. rebar3_erli18n 0.2.0 requires erli18n ~> 0.7.

erli18n 0.8.0 is out on Hex. This one is library-only — the rebar3_erli18n plugin is unchanged at 0.2.0.

The new piece is an optional erlydtl template bridge, erli18n_erlydtl. It lets an erlydtl template translate its {% trans %} / {% blocktrans %} tags through erli18n’s gettext core — contexts, CLDR plurals, per-request locale — while erlydtl keeps ownership of {{ var }} interpolation and auto-escaping. erli18n_erlydtl:translation_fun/1 returns a render-time fun bound to a gettext domain that you pass in erlydtl’s render/2 options; the pure decode/2 maps {% trans %} → gettext, context → pgettext, a counted {% blocktrans %} → ngettext, and context + count → npgettext.

To be straight about the Elixir angle: erlydtl is Erlang’s Django-style template engine, and most Elixir apps reach for EEx/HEEx with Gettext instead — so this is niche here. It matters if you actually render erlydtl templates: a mixed Erlang/Elixir app, or a ported Django project. erli18n has no Elixir dependency; from Elixir you would build the fun with :erli18n_erlydtl.translation_fun(:web) and hand it to :erlydtl’s render options. If you know Gettext, the tags map onto the same four calls Gettext exposes (gettext / pgettext / ngettext / npgettext), and the locale composes with the per-process :erli18n.setlocale(locale) you would set in a Plug — the same spot as Gettext.put_locale/1.

The integration is inverted: erli18n_erlydtl references zero erlydtl functions (erlydtl calls into the fun), so it is not an optional_applications entry and the published package still builds on kernel + stdlib alone — erlydtl is a test-only dependency you pull in only if you use it. A runnable example is in examples/erli18n_erlydtl_demo.