HeuristicTextFilterOptions
Options & Configuration · AiDotNet.Data.Quality
Configuration options for heuristic text quality filtering.
How It Works
Applies simple rule-based filters commonly used for web-crawl cleanup (e.g., C4, CCNet).
Constructors
| Constructor | Summary |
|---|---|
HeuristicTextFilterOptions() |
Properties
| Property | Summary |
|---|---|
bool FilterBoilerplate { get; set; } |
Whether to filter documents containing common boilerplate phrases. |
double MaxAvgWordLength { get; set; } |
Maximum average word length in characters. |
double MaxDigitRatio { get; set; } |
Maximum ratio of digits to total characters. |
double MaxEllipsisLineRatio { get; set; } |
Maximum ratio of lines that end with an ellipsis. |
double MaxSpecialCharRatio { get; set; } |
Maximum ratio of special characters to total characters. |
double MaxUppercaseRatio { get; set; } |
Maximum ratio of uppercase characters to alphabetic characters. |
double MinAvgWordLength { get; set; } |
Minimum average word length in characters. |
double MinPunctuationEndRatio { get; set; } |
Minimum proportion of lines that end with punctuation. |
int MaxWordCount { get; set; } |
Maximum number of words in a document. |
int MinWordCount { get; set; } |
Minimum number of words in a document. |
Methods
| Method | Summary |
|---|---|
void Validate() |
Validates that all option values are within acceptable ranges. |