LanguageIdFilter
Models & Types · AiDotNet.Data.Quality
Filters documents based on detected language using character n-gram profiles.
How It Works
Uses a simple but effective character n-gram frequency approach for language detection. Requires training with reference text for each target language. Based on the Cavnar-Trenkle (1994) text categorization method.
Constructors
| Constructor | Summary |
|---|---|
LanguageIdFilter(LanguageIdFilterOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> Filter(IReadOnlyList<string> documents) |
Filters documents by language, returning indices of documents that should be removed. |
ValueTuple<string, double> DetectLanguage(string text) |
Detects the most likely language of a text. |
void AddLanguageProfile(string languageCode, IReadOnlyList<string> referenceTexts) |
Adds a language profile built from reference text. |