Text - a text analysis library

Based on the positive feedback it seems this project has merit. I didn’t get finished all the work I planned for the weekend but I’m clearing a backlog so I can give this project some greater attention. The inflector is finished (nouns, pronouns, verbs). I’ve started work on adding the Metaphone 2 algorithm but its a slow slog because there is no description of the algorithm I can find - just imperative code. And then some more testing and verification on language detection.

All in all, likely a one week delay.

Thought I’d share the near term roadmap in a little more detail. Feedback is most definitely welcome on the capabilities you would find most useful. Or any areas you’d like to contribute to.

Step 1: Language recognition

Most natural language processing is language dependent. So identifying the source language is important. The primary way of identifying languages is to split the text into n-grams and then perform various statistical analysis of the source text versus the same analysis of a standard corpora in multiple different languages. The Universal Declaration of Human Rights is a standard text published in a lot of languages so this is the corpora I’m using. There are different ways to correlate source text versus a corpora. I am primarily using the algorithms in Language Identification from Text Using N-gram Based Cumulative Frequency Addition.

This is the due now for delivery on 28th June.

Step 2: Text segmentation

No matter what analysis is required, segmenting the text into grapheme clusters, words and sentences is required. This is very language dependent. Elixir’s String.graphemes/1 implements the Unicode segmentation algorithm for grapheme clusters so thats taken care of. Elixir’s String.split/1 implements the Unicode segmentation algorithm for words. String.split/1 is great for a default case but its not sufficient for language-specific segmentation. And we still need sentence segmentation too. Therefore I am implementing the CLDR Segmentation rules which provide language-specific customisation for text segmentation. This is another rules parser (I think so far I have implemented 8 different rules parsers and “compilers” in various parts of the ex_cldr project).

The text segmentation algorithms will be implemented as part of the unicode_string library.

Step 3: Parts of Speech Tagging

Now we have segments of text we can proceed to understanding what is being expressed. The starting point for this is called “parts of speech tagging”. Because I want a good native Elixir implementation that supports a wide variety of languages with good (but not necessarily the absolutely best) tagging I’m using A Rule-based Part-of-Speech and Morphological Tagging Toolkit which provides a fully trained corpora for ~90 languages using the open source data maintained in the Universal Dependencies treebanks. The trained models are maintained in the RDRPOSTagger project which also defines a rules engine that I will implement in Elixir. Another parser/compiler

Step 4: Sentiment Analysis

Now that we have a grammatical breakdown of the target source we can start to identify meaning. Wikipedia says:

A basic task in sentiment analysis is classifying the polarity of a given text at the document, sentence, or feature/aspect level—whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral. Advanced, “beyond polarity” sentiment classification looks, for instance, at emotional states such as “angry”, “sad”, and “happy”.

The implementation approach is not yet defined and feedback and suggestions are warmly welcomed.

Step 5

To be determined. It will take a 2-4 months to get through the first 4 steps so thats plenty of time for feedback and collaboration

text version 2.0 has been published today, just on schedule. In addition the library text_corpus_udhr is also published today - it provides a corpus to support natural language detection.

Language Detection

text contains 3 language classifiers to aid in natural language detection. However it does not include any corpora; these are contained in separate libraries. The available classifiers are:

• Text.Language.Classifier.CommulativeFrequency
• Text.Language.Classifier.NaiveBayesian
• Text.Language.Classifier.RankOrder

Additional classifiers can be added by defining a module that implements the Text.Language.Classifier behaviour.

The library text_corpus_udhr implements the Text.Corpus behaviour for the United National Declaration of Human Rights which is available for download in 423 languages from Unicode.

Examples:

iex> Text.Language.detect "this is some english language thing"
{:ok, "en"}

# Options include `:corpus`, `:vocabulary` and `:classifier`
iex> Text.Language.detect "this is some english language thing", corpus: Text.Corpus.Udhr, vocabulary: Text.Vocabulary.Udhr.Quadgram
{:ok, "en"}

Word Counting

text contains an implementation of word counting that is oriented towards large streams of words rather than discrete strings. Input to Text.Word.word_count/2 can be a String.t, File.Stream.t or Flow.t allowing flexible streaming of text.

English Pluralization

text includes an inflector for the English language that takes an approach based upon An Algorithmic Approach to English Pluralization. See the module Text.Inflect.En and the functions:

• Text.Inflect.En.pluralize/2
• Text.Inflect.En.pluralize_noun/2
• Text.Inflect.En.pluralize_verb/1
• Text.Inflect.En.pluralize_adjective/1

N-Gram generation

The Text.Ngram module supports efficient generation of n-grams of length 2 to 7. See Text.Ngram.ngram/2.

Language detection accuracy

Detection accuracy is reliable at text lengths of 150 characters or more, reasonable at 100 characters and may not be considered acceptable at shorter lengths.

The results are consistent for the range of tested languages with German being a clear exception where the results are unacceptable for now.

• English
• Greek
• Russian
• Spanish
• Finnish
• French
• Icelandic
• Italian
• Japanese
• Simplified Chinese

Further details are contained in the github repo in the analysis directory.

English language with Naive Bayesian classifier

Text.Language.detect/2 with classifier: Text.Classifier.NaiveBayesian and three different vocabularies.

Accuracy for German language detection

German is an exception to the consistent accuracy of most languages and the results are poor. Further analysis is required to understand the underlying cause.

How can I do that?

I would say that you have to create PR in kipcole9/text_corpus_udhr where you put a file, e.g. corpus/udhr/udhr_cze.txt with translation of udhr_eng.txt.

Seems to be there already, right? text_corpus_udhr/corpus/udhr/udhr_ces.txt at master · kipcole9-legacy/text_corpus_udhr · GitHub

Thanks for the interest! Language detection should work at an acceptable level for ~200 languages using the UDHR corpus. Of course you can also contribute additional corpora in a library of your own making as long as it has a module that implements the Text.Corpus behaviour.

Inflection - specifically pluralisation - is on a per-language basis. If you would like to contribute an inflector then a PR with a module called Text.Inflect.<BCP47 language code> that implements a function called pluralize/2 would be “all” thats required. I will define a Text.Inflection behaviour in the 0.3.0 release to make this more clearly defined.

Mea culpa, I was looking for that file but obviously not successfully…

Nothing to apologise for. I need to add a Contributing section to the docs - thankis for the prompt to do so

I think this is cool but i dont have a usecase for it. Keep up the good work