IPruningStrategy<T>

Interfaces · AiDotNet.Interfaces

Interface for pruning strategies that remove unimportant weights to create sparsity.

Known implementations: GradientPruningStrategy<T>, LotteryTicketPruningStrategy<T>, MagnitudePruningStrategy<T>, StructuredPruningStrategy<T>

For Beginners

Pruning removes unnecessary connections from neural networks.

Think of it like pruning a tree - you remove branches that don’t contribute much:

  • Magnitude pruning: Remove smallest weights
  • Gradient pruning: Remove weights with smallest gradients (learning slowly)
  • Structured pruning: Remove entire neurons/filters (cleaner architecture)
  • Movement pruning: Remove weights that don’t change during training
  • Lottery ticket: Find sparse subnetworks that train well from scratch

Sparsity patterns:

  • Unstructured: Random individual weights (flexible but needs sparse libraries)
  • Structured: Entire rows/columns (actual speedup on any hardware)
  • 2:4 Sparsity: 2 zeros per 4 elements (NVIDIA Ampere 2x speedup)
  • N:M Sparsity: N zeros per M elements (customizable)

Pruning can remove 50-99% of weights with minimal accuracy loss!

How It Works

Pruning strategies determine which weights to remove from a neural network to reduce size and computational requirements. This interface supports all data types (Vector, Matrix, Tensor) and multiple sparsity patterns including unstructured, structured, and hardware-optimized formats.

Properties

Property Summary
IReadOnlyList<SparsityPattern> SupportedPatterns { get; } Gets supported sparsity patterns.
bool IsStructured { get; } Gets whether this is structured pruning (removes entire rows/cols/filters).
bool RequiresGradients { get; } Gets whether this strategy requires gradients.
string Name { get; } Gets the name of this pruning strategy.

Methods

Method Summary
IPruningMask<T> Create2to4Mask(Tensor<T> importanceScores) Creates a 2:4 structured sparsity mask (NVIDIA Ampere compatible).
IPruningMask<T> CreateMask(Matrix<T> importanceScores, double targetSparsity) Creates a pruning mask for tensor weights based on target sparsity.
IPruningMask<T> CreateMask(Tensor<T> importanceScores, double targetSparsity) Creates a pruning mask for tensor weights based on target sparsity.
IPruningMask<T> CreateMask(Vector<T> importanceScores, double targetSparsity) Creates a pruning mask for tensor weights based on target sparsity.
IPruningMask<T> CreateNtoMMask(Tensor<T> importanceScores, int n, int m) Creates an N:M structured sparsity mask.
Matrix<T> ComputeImportanceScores(Matrix<T> weights, Matrix<T> gradients) Computes importance scores for tensor weights.
SparseCompressionResult<T> ToSparseFormat(Tensor<T> weights, SparseFormat format) Converts pruned weights to sparse format for efficient storage.
Tensor<T> ComputeImportanceScores(Tensor<T> weights, Tensor<T> gradients) Computes importance scores for tensor weights.
Vector<T> ComputeImportanceScores(Vector<T> weights, Vector<T> gradients) Computes importance scores for tensor weights.
void ApplyPruning(Matrix<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.
void ApplyPruning(Tensor<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.
void ApplyPruning(Vector<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.