I (we, at dscout) definitely have a usecase for nearly all of this work . I hope to contribute in the future, and would love to support the effort financially if you decide to make that possible 
.
sorentwo
Oban Core Team
6
kip
ex_cldr Core Team
Just a little fun addition over coffee this morning - deriving a CLDR locale from natural language. I’ll publish it to hex after I add some tests.
Examples
iex> Cldr.Text.locale_from_text "this is some text that I think will be English"
{:ok, %Cldr.LanguageTag{
backend: MyApp.Cldr,
canonical_locale_name: "en-Latn-US",
cldr_locale_name: "en",
extensions: %{},
gettext_locale_name: nil,
language: "en",
language_subtags: [],
language_variant: nil,
locale: %{},
private_use: [],
rbnf_locale_name: "en",
requested_locale_name: "en",
script: "Latn",
territory: :US,
transform: %{}
}}
iex> german_text = "Wir wohnen in einem kleinen Haus mit einem Garten. Dort können die Kinder ein bisschen spielen. Unser Sohn kommt bald in die Schule, unsere Tochter geht noch eine Zeit lang in den Kindergarten. Meine Kinder sind am Nachmittag zu Hause. So arbeite ich nur halbtags."
iex> Cldr.Text.locale_from_text german_text
{:ok, %Cldr.LanguageTag{
backend: MyApp.Cldr,
canonical_locale_name: "de-Latn-DE-1901",
cldr_locale_name: "de",
extensions: %{},
gettext_locale_name: nil,
language: "de",
language_subtags: [],
language_variant: "1901",
locale: %{},
private_use: [],
rbnf_locale_name: "de",
requested_locale_name: "de-1901",
script: "Latn",
territory: :DE,
transform: %{}
}}
8
rengel
kip
ex_cldr Core Team
Thanks much for the link. I’m a bit challenged reading these imperative implementations for two reasons: (a) such ugly code compared to using pattern matching for most of it as one would in Elixir and (b) as a result, I just want the rules. Megaphone I can find them, but not double.
Maybe I’ll do a basic Metaphone implementation first and at least move forward …
0
tfwright
I’m really interested in using this library in a project of mine, in particular to generate something similar to “word clouds” where common significant words are highlighted. Is that something you are planning on supporting? Please let me know if there’s any part I can help out with.
1
kip
ex_cldr Core Team
I’ll shortly be launching Text, a nascent text analysis library.
Well, instead of shortly, what I should have said is in about 6 years. But fear not:
See the YouTube video if the meme isn’t familiar to you.
A lot has change in the NLP world since 2020 and the new, modernised and thoroughly renovated text 0.3.0 is now available on Hex.
Features
Language identification
Text.Language.Classifier.Fasttext is a pure-Elixir port of lid.176, validated bit-for-bit against fastText’s reference. 176 languages, ~100 µs per prediction with EXLA.
Sentiment, POS, NER
Multilingual sentiment via bundled AFINN lexicons (default) or XLM-RoBERTa through Bumblebee (optional). Part-of-speech tagging and multilingual named-entity recognition via Bumblebee, with mix text.download_models to pre-fetch the weights at deploy time.
Word clouds
A new Text.WordCloud module with six scoring backends — YAKE! by default, plus frequency, RAKE, TextRank, TF-IDF, and a KeyBERT variant for users who configure Bumblebee.
Text.WordCloud.Layout does Wordle-style spiral packing (with :radial and :spiral orientation modes for sunburst and vortex looks), and Text.WordCloud.SVG produces renderer-agnostic output that plays nicely with Color.Palette for coordinated tonal-scale colour ramps. You can see some examples in the guide.
Fundamentals
- String distance and similarity (Levenshtein, Damerau-Levenshtein, Jaro-Winkler, Jaccard, cosine, …)
- Phonetic encoding (Soundex, Metaphone),
- Unicode-aware segmentation, slug generation, TF-IDF and BM25 search,
- Collocation extraction (PMI, log-likelihood), and
- Keyword-in-context concordance.
Also includes bundled stopwords for ~60 languages from stopwords-iso, and optional Snowball stemming via :text_stemmer to consolidate morphological variants in word clouds.
Optional ML
Heavy ML deps (:bumblebee, :exla, KeyBERT) are all optional. Without them, the package still does most of what it does — just without the neural-quality ceiling. Same goes for :color (SVG palettes), :localize (CLDR locale resolution), and :text_stemmer.
6
kip
ex_cldr Core Team
There are two companion libraries also published today:
- Snowball that implements the Snowball language as a cross compiler to Elixir. Its a fun language with its roots in SNOBOL but was designed by Porter specifically to support the implementation of language stemmers. This is not a general purpose language or compiler - there are no conveniences at all. Just a mix task to take
.sblfiles and cross-compile them to Elixir. - text_stemmer which implements all 37 stemming algorithms - validated against their respective conformance suites.
text_stemmer is an optional dependency for text which can be used to tune word clouds.
These two are 100% LLM generated. It took 4 complete days for Claude (Sonnet) to implement. It’s a classic LLM opportunity. Formal specification, canonical implementations, complete conformance testing suites. Prompt the LLM and then go work on something else.
7
kip
ex_cldr Core Team
I’ve published Text 0.4.0 today with seven new NLP modules (all native Elixir, no NIF or ML).
They cover the kinds of preprocessing you might reach for once your sentiment / classification / search pipeline outgrows String.split/1.
This release represents a largely feature complete text library from my perspective. Happy to take feature suggestions though.
A few of the more immediately useful additions in this release:
Text.Clean — pipeline-style normalisation
Whitespace, control characters, smart quotes, mojibake, NFC/NFKC. Composable; defaults are sensible.
iex> Text.Clean.clean("<p>it’s <em>cool</em></p>")
"it's cool"
iex> Text.Clean.collapse_whitespace(" hello \tworld \n")
"hello world"
Text.Truecase — restore casing for ALL-CAPS or lowercased text
POS-aware heuristics for proper nouns, acronyms, and sentence starts. Useful when an upstream system has destroyed the casing (chat logs, OCR, screaming customer feedback).
iex> Text.Truecase.truecase("THE QUICK BROWN FOX JUMPS OVER NEW YORK")
"The quick brown fox jumps over New York"
iex> Text.Truecase.truecase("nasa launched apollo 11 in july 1969.")
"NASA launched Apollo 11 in July 1969."
# Add domain-specific terms once at boot
Text.Truecase.add_terms(["GraphQL", "Phoenix"])
Text.Truecase.truecase("we use phoenix and graphql")
#=> "we use Phoenix and GraphQL"
Text.Emoji — detection, stripping, counting, conversion
Backed by the :unicode package’s emoji property tables, so it recognises every codepoint flagged emoji in the current Unicode release — no shipped JSON.
iex> Text.Emoji.count("Loved it 🤩 read it twice 📚📚")
3
iex> Text.Emoji.demojize("ship it 🚀")
"ship it :rocket:"
iex> Text.Emoji.emojize("ship it :rocket:")
"ship it 🚀"
Text.Hyphenation — Knuth–Liang TeX-pattern hyphenation
Ships en-US patterns baked in (~5 000). Other languages load from any standard hyph-*.tex file.
iex> Text.Hyphenation.hyphenate("hyphenation")
"hy-phen-ation"
iex> Text.Hyphenation.count("supercalifragilisticexpialidocious")
9
# Load German patterns once; thereafter all calls are fast
Text.Hyphenation.load_language(:de, path: "hyph-de-1996.tex")
Text.Hyphenation.hyphenate("Bundesausbildungsförderungsgesetz", language: :de)
#=> "Bun-des-aus-bil-dungs-för-de-rungs-ge-setz"
Text.PII — detect & redact common identifiers
Phone, email, credit-card-shaped digits, IBANs, IPv4/IPv6, US SSN. Pattern-based — fast and deterministic. The right tool for “please don’t paste this into the LLM” preflight; pair with a stricter checker if you need legal-grade accuracy.
iex> Text.PII.detect("Email me at jane@example.com or call (415) 555-0142.")
[%{type: :email, value: "jane@example.com", offset: 12, length: 16}, %{type: :phone, value: "(415) 555-0142", offset: 37, length: 14}]
iex> Text.PII.redact("Card 4111-1111-1111-1111 expires 12/29")
"Card [CREDIT_CARD] expires 12/29"
Text.Spell — Norvig-style spelling suggestions
Edit-distance candidates ranked by frequency in Text.WordFreq (the 30,000-word English frequency table that also ships in 0.4.0).
iex> Text.Spell.correct("speling")
"spelling"
iex> Text.Spell.candidates("teh") |> Enum.take(3)
[%{word: "the", distance: 1, frequency: 6_187_267}, %{word: "tech", distance: 1, frequency: 49_320}, %{word: "ten", distance: 1, frequency: 21_117}]
Text.Summarize — extractive summarisation via TextRank
Sentence-graph TextRank with configurable similarity (:cosine or :jaccard) and target length.
article = """
The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Engineers worked three winters to anchor the central pier on the riverbed. Residents who used to take a 40-minute ferry now make the trip in five. The mayor said the project came in 2 % under budget, a rarity for civic work of this scale.
"""
iex> Text.Summarize.summarize(article, sentences: 2)
"The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Residents who used to take a 40-minute ferry now make the trip in five."
12
kip
ex_cldr Core Team
I’ve published a couple of updates in the last week. I think that pretty much all I need in a text analysis library but I’m more than happy to take feature requests.
[0.6.0] — 2026-05-04
Added
-
Text.Extract— twitter-text-quality URL and email extraction with full UTS #46 IDNA, IANA TLD validation, and UTR #39 single-script defence against homograph attacks. Public API isurls/2,emails/2,all/2,split/2, andautolink/2; options include:require_scheme,:tld_mode,:eai,:strict_idn, and:twitter_quirks. -
Text.Extract.split/2— splits text into an interleaved list of plain-string fragments and validated entity maps, byte-for-byte round-trippable to the original. The building block for custom rendering of extracted URLs/emails into anchors, mentions, badges, or link-preview cards. -
Text.Extract.autolink/2— wraps URLs and emails in HTML<a>anchors, returningPhoenix.HTML.safe()for drop-in Phoenix template use. Display text preserves the original Unicode (bücher.de); thehrefuses Punycode (xn--bcher-kva.de). -
mix text.download_tlds— refreshes the bundled IANA TLD list atpriv/extract/tlds.txt.--diffpreviews added/removed entries;--forceoverwrites unconditionally. -
Text.WordCloud.to_d3_cloud/2— adaptsterms/2output into the[%{text, size}, …]shape consumed by d3-cloud. Supports:linear(default) and:sqrtsizing; shares the:font_size_rangevocabulary withText.WordCloud.Layout.
[0.5.0] — 2026-05-02
Added
-
Text.Phonetic.NYSIIS— New York State Identification and Intelligence System phonetic encoding (Taft, 1970). Designed as a Soundex successor for English personal-name matching; produces pronounceable letter codes rather than digits and is more discriminating than Soundex on common name variations. -
Text.Phonetic.Cologne— Kölner Phonetik (Postel, 1969), the German-language counterpart to Soundex. Optimized for German spelling variants —Müller/Mueller/MullerandMeyer/Mayer/Maier/Meiercollapse to single codes. -
Text.Phonetic.DoubleMetaphone— Lawrence Philips’ Double Metaphone (2000), the de-facto standard for fuzzy English-name matching with non-Anglo origins. Returns a{primary, alternate}code pair so the same Anglicised name can match across multiple plausible pronunciations (e.g.Smith↔Schmidt,Catherine↔Katherine). Handles Germanic, Italian, Spanish, French, Greek, and Slavic patterns. -
match?/2(andmatch?/3where options apply) on everyText.Phonetic.*module for direct equality comparison without manualencode/2 == encode/2boilerplate.Text.Phonetic.DoubleMetaphone.match?/3checks all four primary/alternate combinations. -
Text.Clean.unaccent/1— strip diacritics and fold non-decomposable Latin letters (Þ→Th,ß→ss,Æ→AE,ł→l,đ→d) by delegating toUnicode.Transform.LatinAscii.transform/1. Also exposed as the:unaccentoption onText.Clean.clean/2. -
Text.Distancegains four set-based similarity metrics over character n-grams:jaccard/3,sorensen_dice/3,tanimoto/3(alias forjaccard/3), andcosine/3. All accept an:noption for configurable shingle size (default 2). Operate at the grapheme level for Unicode correctness. -
Text.Inflect.En.singularize/2andText.Inflect.En.singularize_noun/2— invert the existing pluralizer. Combines reverse lookup of Conway’s irregular tables, explicit suffix rules for unambiguous English plural forms (-ies,-shes/-ches/-xes/-zes/-sses), small whitelists for Greek-derived-is/-esplurals (analyses → analysis) and English-usplurals (geniuses → genius), and apluralize/2round-trip search to validate other candidates. -
Text.Readability.dale_chall/2andText.Readability.spache/2— the two classic word-list readability indices, backed by bundled easy-words lists inpriv/readability/(Dale-Chall 2,949 words, Spache 1,063 words; both sourced from the MIT-licensedpy-readability-metricsdistribution of the public-domain originals).statistics/2now also returns:difficult_wordsand:unfamiliar_wordscounts. -
Text.Hyphenationbundles six additional language packs:de-1996,fr,es,it,nl,pt. All loaded at compile time with zero I/O, joining the existingen-uspack. Source: hyph-utf8 upstream; per-file licenses (MIT/X11/BSD/LPPL) are preserved in each.texheader. -
Text.WordFreqbundles six additional frequency tables at the same top-30,000 cap as English:de,fr,es,it,nl,pt. Source: Hermit Dave’s MIT-licensed FrequencyWords OpenSubtitles 2018 corpus. -
Text.Emoji.sentiment/1andText.Emoji.text_sentiment/1— per-emoji and aggregate sentiment scoring backed by the bundled Emoji Sentiment Ranking v1.0 (Kralj Novak et al., 2015; ~750 emoji with negative/neutral/positive proportions and an aggregate score in[-1.0, 1.0]). Aggregate scoring is occurrence-weighted to match the original paper.
6