Text - a text analysis library

I’ve published a couple of updates in the last week. I think that pretty much all I need in a text analysis library but I’m more than happy to take feature requests.

[0.6.0] — 2026-05-04

Added

• Text.Extract — twitter-text-quality URL and email extraction with full UTS #46 IDNA, IANA TLD validation, and UTR #39 single-script defence against homograph attacks. Public API is urls/2, emails/2, all/2, split/2, and autolink/2; options include :require_scheme, :tld_mode, :eai, :strict_idn, and :twitter_quirks.

• Text.Extract.split/2 — splits text into an interleaved list of plain-string fragments and validated entity maps, byte-for-byte round-trippable to the original. The building block for custom rendering of extracted URLs/emails into anchors, mentions, badges, or link-preview cards.

• Text.Extract.autolink/2 — wraps URLs and emails in HTML <a> anchors, returning Phoenix.HTML.safe() for drop-in Phoenix template use. Display text preserves the original Unicode (bücher.de); the href uses Punycode (xn--bcher-kva.de).

• mix text.download_tlds — refreshes the bundled IANA TLD list at priv/extract/tlds.txt. --diff previews added/removed entries; --force overwrites unconditionally.

• Text.WordCloud.to_d3_cloud/2 — adapts terms/2 output into the [%{text, size}, …] shape consumed by d3-cloud. Supports :linear (default) and :sqrt sizing; shares the :font_size_range vocabulary with Text.WordCloud.Layout.

[0.5.0] — 2026-05-02

Added

• Text.Phonetic.NYSIIS — New York State Identification and Intelligence System phonetic encoding (Taft, 1970). Designed as a Soundex successor for English personal-name matching; produces pronounceable letter codes rather than digits and is more discriminating than Soundex on common name variations.

• Text.Phonetic.Cologne — Kölner Phonetik (Postel, 1969), the German-language counterpart to Soundex. Optimized for German spelling variants — Müller / Mueller / Muller and Meyer / Mayer / Maier / Meier collapse to single codes.

• Text.Phonetic.DoubleMetaphone — Lawrence Philips’ Double Metaphone (2000), the de-facto standard for fuzzy English-name matching with non-Anglo origins. Returns a {primary, alternate} code pair so the same Anglicised name can match across multiple plausible pronunciations (e.g. Smith ↔ Schmidt, Catherine ↔ Katherine). Handles Germanic, Italian, Spanish, French, Greek, and Slavic patterns.

• match?/2 (and match?/3 where options apply) on every Text.Phonetic.* module for direct equality comparison without manual encode/2 == encode/2 boilerplate. Text.Phonetic.DoubleMetaphone.match?/3 checks all four primary/alternate combinations.

• Text.Clean.unaccent/1 — strip diacritics and fold non-decomposable Latin letters (Þ → Th, ß → ss, Æ → AE, ł → l, đ → d) by delegating to Unicode.Transform.LatinAscii.transform/1. Also exposed as the :unaccent option on Text.Clean.clean/2.

• Text.Distance gains four set-based similarity metrics over character n-grams: jaccard/3, sorensen_dice/3, tanimoto/3 (alias for jaccard/3), and cosine/3. All accept an :n option for configurable shingle size (default 2). Operate at the grapheme level for Unicode correctness.

• Text.Inflect.En.singularize/2 and Text.Inflect.En.singularize_noun/2 — invert the existing pluralizer. Combines reverse lookup of Conway’s irregular tables, explicit suffix rules for unambiguous English plural forms (-ies, -shes/-ches/-xes/-zes/-sses), small whitelists for Greek-derived -is/-es plurals (analyses → analysis) and English -us plurals (geniuses → genius), and a pluralize/2 round-trip search to validate other candidates.

• Text.Readability.dale_chall/2 and Text.Readability.spache/2 — the two classic word-list readability indices, backed by bundled easy-words lists in priv/readability/ (Dale-Chall 2,949 words, Spache 1,063 words; both sourced from the MIT-licensed py-readability-metrics distribution of the public-domain originals). statistics/2 now also returns :difficult_words and :unfamiliar_words counts.

• Text.Hyphenation bundles six additional language packs: de-1996, fr, es, it, nl, pt. All loaded at compile time with zero I/O, joining the existing en-us pack. Source: hyph-utf8 upstream; per-file licenses (MIT/X11/BSD/LPPL) are preserved in each .tex header.

• Text.WordFreq bundles six additional frequency tables at the same top-30,000 cap as English: de, fr, es, it, nl, pt. Source: Hermit Dave’s MIT-licensed FrequencyWords OpenSubtitles 2018 corpus.

• Text.Emoji.sentiment/1 and Text.Emoji.text_sentiment/1 — per-emoji and aggregate sentiment scoring backed by the bundled Emoji Sentiment Ranking v1.0 (Kralj Novak et al., 2015; ~750 emoji with negative/neutral/positive proportions and an aggregate score in [-1.0, 1.0]). Aggregate scoring is occurrence-weighted to match the original paper.

I’ve published Text 0.4.0 today with seven new NLP modules (all native Elixir, no NIF or ML).

They cover the kinds of preprocessing you might reach for once your sentiment / classification / search pipeline outgrows String.split/1.

This release represents a largely feature complete text library from my perspective. Happy to take feature suggestions though.

A few of the more immediately useful additions in this release:

Text.Clean — pipeline-style normalisation

Whitespace, control characters, smart quotes, mojibake, NFC/NFKC. Composable; defaults are sensible.

iex> Text.Clean.clean("<p>it’s   <em>cool</em></p>")
"it's cool"
iex> Text.Clean.collapse_whitespace("  hello \tworld  \n")
"hello world"

Text.Truecase — restore casing for ALL-CAPS or lowercased text

POS-aware heuristics for proper nouns, acronyms, and sentence starts. Useful when an upstream system has destroyed the casing (chat logs, OCR, screaming customer feedback).

iex> Text.Truecase.truecase("THE QUICK BROWN FOX JUMPS OVER NEW YORK")
"The quick brown fox jumps over New York"
iex> Text.Truecase.truecase("nasa launched apollo 11 in july 1969.")
"NASA launched Apollo 11 in July 1969."

# Add domain-specific terms once at boot
Text.Truecase.add_terms(["GraphQL", "Phoenix"])
Text.Truecase.truecase("we use phoenix and graphql")
#=> "we use Phoenix and GraphQL"

Text.Emoji — detection, stripping, counting, conversion

Backed by the :unicode package’s emoji property tables, so it recognises every codepoint flagged emoji in the current Unicode release — no shipped JSON.

iex> Text.Emoji.count("Loved it 🤩 read it twice 📚📚")
3
iex> Text.Emoji.demojize("ship it 🚀")
"ship it :rocket:"
iex> Text.Emoji.emojize("ship it :rocket:")
"ship it 🚀"

Text.Hyphenation — Knuth–Liang TeX-pattern hyphenation

Ships en-US patterns baked in (~5 000). Other languages load from any standard hyph-*.tex file.

iex> Text.Hyphenation.hyphenate("hyphenation")
"hy-phen-ation"
iex> Text.Hyphenation.count("supercalifragilisticexpialidocious")
9
# Load German patterns once; thereafter all calls are fast
Text.Hyphenation.load_language(:de, path: "hyph-de-1996.tex")
Text.Hyphenation.hyphenate("Bundesausbildungsförderungsgesetz", language: :de)
#=> "Bun-des-aus-bil-dungs-för-de-rungs-ge-setz"

Text.PII — detect & redact common identifiers

Phone, email, credit-card-shaped digits, IBANs, IPv4/IPv6, US SSN. Pattern-based — fast and deterministic. The right tool for “please don’t paste this into the LLM” preflight; pair with a stricter checker if you need legal-grade accuracy.

iex> Text.PII.detect("Email me at jane@example.com or call (415) 555-0142.")
[%{type: :email, value: "jane@example.com",  offset: 12, length: 16},  %{type: :phone, value: "(415) 555-0142",    offset: 37, length: 14}]
iex> Text.PII.redact("Card 4111-1111-1111-1111 expires 12/29")
"Card [CREDIT_CARD] expires 12/29"

Text.Spell — Norvig-style spelling suggestions

Edit-distance candidates ranked by frequency in Text.WordFreq (the 30,000-word English frequency table that also ships in 0.4.0).

iex> Text.Spell.correct("speling")
"spelling"
iex> Text.Spell.candidates("teh") |> Enum.take(3)
[%{word: "the",  distance: 1, frequency: 6_187_267},  %{word: "tech", distance: 1, frequency:    49_320},  %{word: "ten",  distance: 1, frequency:    21_117}]

Text.Summarize — extractive summarisation via TextRank

Sentence-graph TextRank with configurable similarity (:cosine or :jaccard) and target length.

article = """
The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Engineers worked three winters to anchor the central pier on the riverbed. Residents who used to take a 40-minute ferry now make the trip in five. The mayor said the project came in 2 % under budget, a rarity for civic work of this scale.
"""
iex> Text.Summarize.summarize(article, sentences: 2)
"The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Residents who used to take a 40-minute ferry now make the trip in five."

There are two companion libraries also published today:

• Snowball that implements the Snowball language as a cross compiler to Elixir. Its a fun language with its roots in SNOBOL but was designed by Porter specifically to support the implementation of language stemmers. This is not a general purpose language or compiler - there are no conveniences at all. Just a mix task to take .sbl files and cross-compile them to Elixir.
• text_stemmer which implements all 37 stemming algorithms - validated against their respective conformance suites.

text_stemmer is an optional dependency for text which can be used to tune word clouds.

These two are 100% LLM generated. It took 4 complete days for Claude (Sonnet) to implement. It’s a classic LLM opportunity. Formal specification, canonical implementations, complete conformance testing suites. Prompt the LLM and then go work on something else.

I’ll shortly be launching Text, a nascent text analysis library.

Well, instead of shortly, what I should have said is in about 6 years. But fear not:

Its-not-dead-its-resting-meme-generator-e1597175206791550×309 43.5 KB

See the YouTube video if the meme isn’t familiar to you.

A lot has change in the NLP world since 2020 and the new, modernised and thoroughly renovated text 0.3.0 is now available on Hex.

Features

Language identification

Text.Language.Classifier.Fasttext is a pure-Elixir port of lid.176, validated bit-for-bit against fastText’s reference. 176 languages, ~100 µs per prediction with EXLA.

Sentiment, POS, NER

Multilingual sentiment via bundled AFINN lexicons (default) or XLM-RoBERTa through Bumblebee (optional). Part-of-speech tagging and multilingual named-entity recognition via Bumblebee, with mix text.download_models to pre-fetch the weights at deploy time.

Word clouds

A new Text.WordCloud module with six scoring backends — YAKE! by default, plus frequency, RAKE, TextRank, TF-IDF, and a KeyBERT variant for users who configure Bumblebee.

Text.WordCloud.Layout does Wordle-style spiral packing (with :radial and :spiral orientation modes for sunburst and vortex looks), and Text.WordCloud.SVG produces renderer-agnostic output that plays nicely with Color.Palette for coordinated tonal-scale colour ramps. You can see some examples in the guide.

Fundamentals

• String distance and similarity (Levenshtein, Damerau-Levenshtein, Jaro-Winkler, Jaccard, cosine, …)
• Phonetic encoding (Soundex, Metaphone),
• Unicode-aware segmentation, slug generation, TF-IDF and BM25 search,
• Collocation extraction (PMI, log-likelihood), and
• Keyword-in-context concordance.

Also includes bundled stopwords for ~60 languages from stopwords-iso, and optional Snowball stemming via :text_stemmer to consolidate morphological variants in word clouds.

Optional ML

Heavy ML deps (:bumblebee, :exla, KeyBERT) are all optional. Without them, the package still does most of what it does — just without the neural-quality ceiling. Same goes for :color (SVG palettes), :localize (CLDR locale resolution), and :text_stemmer.

I’m really interested in using this library in a project of mine, in particular to generate something similar to “word clouds” where common significant words are highlighted. Is that something you are planning on supporting? Please let me know if there’s any part I can help out with.

Thanks much for the link. I’m a bit challenged reading these imperative implementations for two reasons: (a) such ugly code compared to using pattern matching for most of it as one would in Elixir and (b) as a result, I just want the rules. Megaphone I can find them, but not double.

Maybe I’ll do a basic Metaphone implementation first and at least move forward …

Just stumbled upon this post. In case you didn’t konw:

Just a little fun addition over coffee this morning - deriving a CLDR locale from natural language. I’ll publish it to hex after I add some tests.

Examples

iex> Cldr.Text.locale_from_text "this is some text that I think will be English"
{:ok,                                                                                                                            %Cldr.LanguageTag{   
   backend: MyApp.Cldr,
   canonical_locale_name: "en-Latn-US",
   cldr_locale_name: "en",
   extensions: %{},
   gettext_locale_name: nil,
   language: "en",
   language_subtags: [],
   language_variant: nil,
   locale: %{},
   private_use: [],
   rbnf_locale_name: "en",
   requested_locale_name: "en",
   script: "Latn",
   territory: :US,
   transform: %{}
 }}

iex> german_text = "Wir wohnen in einem kleinen Haus mit einem Garten. Dort können die Kinder ein bisschen spielen. Unser Sohn kommt bald in die Schule, unsere Tochter geht noch eine Zeit lang in den Kindergarten. Meine Kinder sind am Nachmittag zu Hause. So arbeite ich nur halbtags."
iex> Cldr.Text.locale_from_text german_text
{:ok,                                                                                                                            %Cldr.LanguageTag{   
   backend: MyApp.Cldr,
   canonical_locale_name: "de-Latn-DE-1901",
   cldr_locale_name: "de",
   extensions: %{},
   gettext_locale_name: nil,
   language: "de",
   language_subtags: [],
   language_variant: "1901",
   locale: %{},
   private_use: [],
   rbnf_locale_name: "de",
   requested_locale_name: "de-1901",
   script: "Latn",
   territory: :DE,
   transform: %{}
 }}

I (we, at dscout) definitely have a usecase for nearly all of this work . I hope to contribute in the future, and would love to support the effort financially if you decide to make that possible .

I think this is cool but i dont have a usecase for it. Keep up the good work