HeuristicTextFilter

Models & Types · AiDotNet.Data.Quality

Filters text documents using simple heuristic rules for quality assessment.

How It Works

Applies rule-based checks common in web-crawl data cleaning (C4, CCNet, OSCAR). Checks word count, character ratios, punctuation, and boilerplate phrases. Fast and requires no training data.

Constructors

Constructor Summary
HeuristicTextFilter(HeuristicTextFilterOptions options)

Methods

Method Summary
HashSet<int> Filter(IReadOnlyList<string> documents) Filters documents, returning indices of documents that should be removed.
bool PassesFilter(string text) Evaluates a single document against all heuristic rules.