LanguageIdFilter

Models & Types · AiDotNet.Data.Quality

Filters documents based on detected language using character n-gram profiles.

How It Works

Uses a simple but effective character n-gram frequency approach for language detection. Requires training with reference text for each target language. Based on the Cavnar-Trenkle (1994) text categorization method.

Constructors

Constructor Summary
LanguageIdFilter(LanguageIdFilterOptions options)

Methods

Method Summary
HashSet<int> Filter(IReadOnlyList<string> documents) Filters documents by language, returning indices of documents that should be removed.
ValueTuple<string, double> DetectLanguage(string text) Detects the most likely language of a text.
void AddLanguageProfile(string languageCode, IReadOnlyList<string> referenceTexts) Adds a language profile built from reference text.