HeuristicTextFilter
Models & Types · AiDotNet.Data.Quality
Filters text documents using simple heuristic rules for quality assessment.
How It Works
Applies rule-based checks common in web-crawl data cleaning (C4, CCNet, OSCAR). Checks word count, character ratios, punctuation, and boilerplate phrases. Fast and requires no training data.
Constructors
| Constructor | Summary |
|---|---|
HeuristicTextFilter(HeuristicTextFilterOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> Filter(IReadOnlyList<string> documents) |
Filters documents, returning indices of documents that should be removed. |
bool PassesFilter(string text) |
Evaluates a single document against all heuristic rules. |