FactorTransferDistillationStrategy<T>

Models & Types · AiDotNet.KnowledgeDistillation.Strategies

Factor transfer distillation that transfers knowledge through factorized representations.

Inheritance: DistillationStrategyBase<T> → FactorTransferDistillationStrategy<T>

How It Works

For Production Use: This strategy implements factor transfer from the paper “Paraphrasing Complex Network: Network Compression via Factor Transfer” (Kim et al., 2018). Instead of directly matching high-dimensional representations, it factorizes them into lower-dimensional factors and matches these factors.

Key Concept: High-dimensional neural network representations contain redundancy. Factor transfer extracts the essential information by:

  1. Decomposing representations into factors (low-rank approximation)
  2. Matching student factors to teacher factors
  3. Reducing noise and focusing on important dimensions

Why This Works:

  • Removes redundancy: Raw features have correlated dimensions
  • Focuses on variance: Principal components capture most information
  • More robust: Less sensitive to noise in individual neurons
  • Compressed transfer: Fewer dimensions to match

Implementation: We provide multiple factorization methods:

  • LowRankApproximation: Use truncated decomposition (similar to PCA)
  • NuclearNormMatching: Match based on singular values (nuclear norm)
  • FactorMatching: Paraphraser network approach with translator

Mathematical Foundation: For feature matrix F ∈ ℝ^(n×d), decompose as F ≈ USV^T where:

  • U ∈ ℝ^(n×k): Samples in factor space
  • S ∈ ℝ^(k×k): Singular values (importance weights)
  • V ∈ ℝ^(d×k): Feature loadings

Match student and teacher factors using MSE or Frobenius norm.

Research Basis:

  • Paraphrasing Complex Network (Kim et al., 2018)
  • FitNets: Hints for Thin Deep Nets (Romero et al., 2015)
  • Attention Transfer (Zagoruyko & Komodakis, 2017)

Constructors

Constructor Summary
FactorTransferDistillationStrategy(double factorWeight, FactorMode mode, int numFactors, bool normalizeFactors, double temperature, double alpha) Creates a factor transfer strategy with a secure, unseeded generator for the power-iteration starts.
FactorTransferDistillationStrategy(double factorWeight, FactorMode mode, int numFactors, bool normalizeFactors, double temperature, double alpha, int? seed) Creates a factor transfer strategy whose power-iteration starts are drawn from a seeded generator.

Methods

Method Summary
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch)
T ComputeFactorLoss(Vector<T>[] studentFeatures, Vector<T>[] teacherFeatures) Computes factor transfer loss by matching factorized representations.
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch)