FactorTransferDistillationStrategy<T>
Models & Types · AiDotNet.KnowledgeDistillation.Strategies
Factor transfer distillation that transfers knowledge through factorized representations.
Inheritance: DistillationStrategyBase<T> → FactorTransferDistillationStrategy<T>
How It Works
For Production Use: This strategy implements factor transfer from the paper “Paraphrasing Complex Network: Network Compression via Factor Transfer” (Kim et al., 2018). Instead of directly matching high-dimensional representations, it factorizes them into lower-dimensional factors and matches these factors.
Key Concept: High-dimensional neural network representations contain redundancy. Factor transfer extracts the essential information by:
- Decomposing representations into factors (low-rank approximation)
- Matching student factors to teacher factors
- Reducing noise and focusing on important dimensions
Why This Works:
- Removes redundancy: Raw features have correlated dimensions
- Focuses on variance: Principal components capture most information
- More robust: Less sensitive to noise in individual neurons
- Compressed transfer: Fewer dimensions to match
Implementation: We provide multiple factorization methods:
- LowRankApproximation: Use truncated decomposition (similar to PCA)
- NuclearNormMatching: Match based on singular values (nuclear norm)
- FactorMatching: Paraphraser network approach with translator
Mathematical Foundation: For feature matrix F ∈ ℝ^(n×d), decompose as F ≈ USV^T where:
- U ∈ ℝ^(n×k): Samples in factor space
- S ∈ ℝ^(k×k): Singular values (importance weights)
- V ∈ ℝ^(d×k): Feature loadings
Match student and teacher factors using MSE or Frobenius norm.
Research Basis:
- Paraphrasing Complex Network (Kim et al., 2018)
- FitNets: Hints for Thin Deep Nets (Romero et al., 2015)
- Attention Transfer (Zagoruyko & Komodakis, 2017)
Constructors
| Constructor | Summary |
|---|---|
FactorTransferDistillationStrategy(double factorWeight, FactorMode mode, int numFactors, bool normalizeFactors, double temperature, double alpha) |
Creates a factor transfer strategy with a secure, unseeded generator for the power-iteration starts. |
FactorTransferDistillationStrategy(double factorWeight, FactorMode mode, int numFactors, bool normalizeFactors, double temperature, double alpha, int? seed) |
Creates a factor transfer strategy whose power-iteration starts are drawn from a seeded generator. |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
|
T ComputeFactorLoss(Vector<T>[] studentFeatures, Vector<T>[] teacherFeatures) |
Computes factor transfer loss by matching factorized representations. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |