HeuristicTextFilterOptions

Options & Configuration · AiDotNet.Data.Quality

Configuration options for heuristic text quality filtering.

How It Works

Applies simple rule-based filters commonly used for web-crawl cleanup (e.g., C4, CCNet).

Constructors

Constructor Summary
HeuristicTextFilterOptions()

Properties

Property Summary
bool FilterBoilerplate { get; set; } Whether to filter documents containing common boilerplate phrases.
double MaxAvgWordLength { get; set; } Maximum average word length in characters.
double MaxDigitRatio { get; set; } Maximum ratio of digits to total characters.
double MaxEllipsisLineRatio { get; set; } Maximum ratio of lines that end with an ellipsis.
double MaxSpecialCharRatio { get; set; } Maximum ratio of special characters to total characters.
double MaxUppercaseRatio { get; set; } Maximum ratio of uppercase characters to alphabetic characters.
double MinAvgWordLength { get; set; } Minimum average word length in characters.
double MinPunctuationEndRatio { get; set; } Minimum proportion of lines that end with punctuation.
int MaxWordCount { get; set; } Maximum number of words in a document.
int MinWordCount { get; set; } Minimum number of words in a document.

Methods

Method Summary
void Validate() Validates that all option values are within acceptable ranges.