PerplexityFilter

Models & Types · AiDotNet.Data.Quality

Filters documents based on perplexity scores from a simple n-gram language model.

How It Works

Perplexity measures how “surprised” a language model is by a document. Very high perplexity indicates gibberish or foreign language text. Very low perplexity may indicate boilerplate or repetitive content. Commonly used in C4 and CCNet data cleaning pipelines.

Constructors

Constructor Summary
PerplexityFilter(PerplexityFilterOptions options)

Methods

Method Summary
HashSet<int> Filter(IReadOnlyList<string> documents) Filters documents by perplexity, returning indices of documents that should be removed.
double ComputePerplexity(string text) Computes the perplexity of a document under the trained language model.
void Train(IReadOnlyList<string> referenceDocuments) Trains the n-gram language model on reference text.