DataPruner
Models & Types · AiDotNet.Data.Quality
Prunes (removes) training samples based on difficulty/importance scores.
How It Works
Data pruning removes easy or redundant samples to reduce training set size while preserving model quality. Supports multiple scoring strategies: confidence-based, forgetting events, EL2N, and GraNd scores. Requires per-sample training signals collected during a warmup phase.
Constructors
| Constructor | Summary |
|---|---|
DataPruner(DataPrunerOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> Prune(double[] scores) |
Identifies samples to prune using the configured strategy. |
HashSet<int> PruneByConfidence(double[] confidenceScores) |
Identifies samples to prune based on confidence scores. |
HashSet<int> PruneByEL2N(double[] el2nScores) |
Identifies samples to prune based on EL2N (Error L2 Norm) scores. |
HashSet<int> PruneByForgetting(int[] forgettingCounts) |
Identifies samples to prune based on forgetting event counts. |
HashSet<int> PruneByGraNd(double[] grandScores) |
Identifies samples to prune based on GraNd (Gradient Norm) scores. |