DataPruner

Models & Types · AiDotNet.Data.Quality

Prunes (removes) training samples based on difficulty/importance scores.

How It Works

Data pruning removes easy or redundant samples to reduce training set size while preserving model quality. Supports multiple scoring strategies: confidence-based, forgetting events, EL2N, and GraNd scores. Requires per-sample training signals collected during a warmup phase.

Constructors

Constructor Summary
DataPruner(DataPrunerOptions options)

Methods

Method Summary
HashSet<int> Prune(double[] scores) Identifies samples to prune using the configured strategy.
HashSet<int> PruneByConfidence(double[] confidenceScores) Identifies samples to prune based on confidence scores.
HashSet<int> PruneByEL2N(double[] el2nScores) Identifies samples to prune based on EL2N (Error L2 Norm) scores.
HashSet<int> PruneByForgetting(int[] forgettingCounts) Identifies samples to prune based on forgetting event counts.
HashSet<int> PruneByGraNd(double[] grandScores) Identifies samples to prune based on GraNd (Gradient Norm) scores.