GradientPruningStrategy<T>
Models & Types · AiDotNet.Pruning
Prunes weights based on gradient magnitude (sensitivity).
Implements: IPruningStrategy<T>
For Beginners
This strategy removes connections that don’t learn much.
Think of it like identifying which team members contribute to a project:
- High gradient = This weight changes a lot during training, it’s learning something important
- Low gradient = This weight barely changes, it’s not contributing much to learning
The importance score is calculated as |weight × gradient|:
- If a weight is large BUT has tiny gradients, it might not be doing much
- If a weight is learning slowly (small gradient), removing it won’t hurt performance
This is smarter than magnitude-based pruning because it considers learning dynamics, not just weight size. However, it requires gradient information from training.
Example:
- Weight = 0.5, Gradient = 0.001 → Importance = |0.5 × 0.001| = 0.0005 (low, prune it)
- Weight = 0.3, Gradient = 0.9 → Importance = |0.3 × 0.9| = 0.27 (high, keep it)
How It Works
Gradient-based pruning uses gradient information to determine weight importance. Weights with small gradients have little impact on the loss function and can be safely removed. This approach considers both the weight value and how much it affects learning.
Constructors
| Constructor | Summary |
|---|---|
GradientPruningStrategy() |
Initializes a new instance of GradientPruningStrategy. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<SparsityPattern> SupportedPatterns { get; } |
Gets supported sparsity patterns. |
bool IsStructured { get; } |
Gets whether this is structured pruning (false for gradient-based). |
bool RequiresGradients { get; } |
Gets whether this strategy requires gradients (true for gradient-based). |
string Name { get; } |
Gets the name of this pruning strategy. |
Methods
| Method | Summary |
|---|---|
IPruningMask<T> Create2to4Mask(Tensor<T> importanceScores) |
Creates a 2:4 structured sparsity mask (NVIDIA Ampere compatible). |
IPruningMask<T> CreateMask(Matrix<T> importanceScores, double targetSparsity) |
Creates a pruning mask for tensor weights based on target sparsity. |
IPruningMask<T> CreateMask(Tensor<T> importanceScores, double targetSparsity) |
Creates a pruning mask for tensor weights based on target sparsity. |
IPruningMask<T> CreateMask(Vector<T> importanceScores, double targetSparsity) |
Creates a pruning mask for tensor weights based on target sparsity. |
IPruningMask<T> CreateNtoMMask(Tensor<T> importanceScores, int n, int m) |
Creates an N:M structured sparsity mask. |
Matrix<T> ComputeImportanceScores(Matrix<T> weights, Matrix<T> gradients) |
Computes importance scores for tensor weights. |
SparseCompressionResult<T> ToSparseFormat(Tensor<T> weights, SparseFormat format) |
Converts pruned weights to sparse format for efficient storage. |
SparseCompressionResult<T> ToSparseFormat(Tensor<T> weights, SparseFormat format, int n, int m) |
Converts pruned weights to N:M structured sparse format for efficient storage. |
Tensor<T> ComputeImportanceScores(Tensor<T> weights, Tensor<T> gradients) |
Computes importance scores for tensor weights. |
Vector<T> ComputeImportanceScores(Vector<T> weights, Vector<T> gradients) |
Computes importance scores for tensor weights. |
void ApplyPruning(Matrix<T> weights, IPruningMask<T> mask) |
Applies pruning mask to tensor weights in-place. |
void ApplyPruning(Tensor<T> weights, IPruningMask<T> mask) |
Applies pruning mask to tensor weights in-place. |
void ApplyPruning(Vector<T> weights, IPruningMask<T> mask) |
Applies pruning mask to tensor weights in-place. |