LotteryTicketPruningStrategy<T>

Models & Types · AiDotNet.Pruning

Implements the Lottery Ticket Hypothesis (Frankle & Carbin, 2019).

Implements: IPruningStrategy<T>

For Beginners

This strategy is based on a fascinating discovery in neural networks!

The Lottery Ticket Hypothesis says: “Inside every large neural network, there’s a smaller ‘winning lottery ticket’ network that could have achieved the same performance if trained from the start.”

The analogy: Imagine you’re building a team:

  • You start with 100 people (full network)
  • After working together, you realize only 20 people did most of the work
  • If you had started with just those 20 people from day one, you’d achieve the same results!

How it works:

  1. Train the network to completion
  2. Find which weights became large (important)
  3. Create a mask keeping only those weights
  4. Reset the KEPT weights to their original random initialization
  5. Retrain with the mask - you’ll match the original accuracy!

This is different from regular pruning because:

  • Regular pruning: Train → Prune → Fine-tune
  • Lottery ticket: Train → Prune → Reset to init → Retrain from scratch

Why this matters:

  • Shows that the structure (which connections) matters more than learned values
  • Enables training sparse networks from scratch
  • Challenges assumptions about why neural networks work

Example workflow:

  1. Initialize network with random weights W₀
  2. Train to get final weights W_final
  3. Create mask M based on |W_final| (keep largest 30%)
  4. Reset: W = W₀ ⊙ M (original weights, masked)
  5. Retrain this sparse network - it matches full network performance!

How It Works

The Lottery Ticket Hypothesis states that dense neural networks contain sparse subnetworks (winning tickets) that, when trained in isolation from initialization, can match the performance of the original network. This strategy finds these winning tickets through iterative pruning and resetting weights to their initial values.

Constructors

Constructor Summary
LotteryTicketPruningStrategy(int iterativeRounds) Creates a new lottery ticket pruning strategy.

Properties

Property Summary
IReadOnlyList<SparsityPattern> SupportedPatterns { get; } Gets supported sparsity patterns (unstructured and N:M patterns).
bool IsStructured { get; } Gets whether this is structured pruning (false for lottery ticket).
bool RequiresGradients { get; } Gets whether this strategy requires gradients (false for lottery ticket).
string Name { get; } Gets the name of this pruning strategy.

Methods

Method Summary
IPruningMask<T> Create2to4Mask(Tensor<T> importanceScores) Creates a 2:4 structured sparsity mask (NVIDIA Ampere compatible).
IPruningMask<T> CreateMask(Matrix<T> importanceScores, double targetSparsity) Creates a pruning mask using iterative magnitude pruning for tensors.
IPruningMask<T> CreateMask(Tensor<T> importanceScores, double targetSparsity) Creates a pruning mask using iterative magnitude pruning for tensors.
IPruningMask<T> CreateMask(Vector<T> importanceScores, double targetSparsity) Creates a pruning mask using iterative magnitude pruning for tensors.
IPruningMask<T> CreateNtoMMask(Tensor<T> importanceScores, int n, int m) Creates an N:M structured sparsity mask.
Matrix<T> ComputeImportanceScores(Matrix<T> weights, Matrix<T> gradients) Computes importance scores using magnitude-based scoring for tensors.
Matrix<T> GetInitialWeights(string layerName) Gets the stored initial weights for a layer.
SparseCompressionResult<T> ToSparseFormat(Tensor<T> weights, SparseFormat format) Converts pruned weights to sparse format for efficient storage.
SparseCompressionResult<T> ToSparseFormat(Tensor<T> weights, SparseFormat format, int n, int m) Converts pruned weights to N:M structured sparse format for efficient storage.
Tensor<T> ComputeImportanceScores(Tensor<T> weights, Tensor<T> gradients) Computes importance scores using magnitude-based scoring for tensors.
Vector<T> ComputeImportanceScores(Vector<T> weights, Vector<T> gradients) Computes importance scores using magnitude-based scoring for tensors.
void ApplyPruning(Matrix<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.
void ApplyPruning(Tensor<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.
void ApplyPruning(Vector<T> weights, IPruningMask<T> mask) Applies pruning mask to tensor weights in-place.
void ResetToInitialWeights(string layerName, Matrix<T> weights, IPruningMask<T> mask) Resets pruned weights to their initial values (key step in lottery ticket).
void StoreInitialWeights(string layerName, Matrix<T> weights) Stores initial weights before training (critical for lottery ticket).