VariationalDistillationStrategy<T>

Models & Types · AiDotNet.KnowledgeDistillation.Strategies

Variational distillation based on variational inference principles and information theory.

Inheritance: DistillationStrategyBase<T> → VariationalDistillationStrategy<T>

How It Works

For Production Use: This strategy applies variational inference to knowledge distillation, treating representations as distributions rather than point estimates. It implements concepts from Variational Information Bottleneck (VIB) and Variational Autoencoders (VAE) for distillation.

Key Concept: Instead of matching point predictions, we model representations as probability distributions (typically Gaussian) and match these distributions. This captures uncertainty and enables:

  1. Latent space alignment - Match distributions in hidden layers
  2. Information bottleneck - Compress while preserving task-relevant information
  3. Uncertainty quantification - Transfer confidence estimates

Implementation: We provide three variational modes:

  • ELBO: Match Evidence Lower Bound (reconstruction + KL)
  • InformationBottleneck: Minimize I(Z;X) while maximizing I(Z;Y) where Z is representation
  • LatentSpaceKL: Match KL divergence in latent space between teacher and student

Mathematical Foundation: For Gaussian distributions N(μ,σ²), the KL divergence is: KL(P||Q) = log(σ_q/σ_p) + (σ_p² + (μ_p - μ_q)²)/(2σ_q²) - 1/2

The VIB objective: min I(X;Z) - βI(Z;Y) where β controls the trade-off.

Research Basis: Based on:

  • Variational Information Bottleneck (Alemi et al., 2017)
  • Variational Knowledge Distillation (Ahn et al., 2019)
  • Bayesian Dark Knowledge (Balan et al., 2015)

Constructors

Constructor Summary
VariationalDistillationStrategy(double variationalWeight, VariationalMode mode, double betaIB, double temperature, double alpha)

Methods

Method Summary
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch)
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch)
T ComputeVariationalLoss(Vector<T> studentMean, Vector<T> studentLogVar, Vector<T> teacherMean, Vector<T> teacherLogVar) Computes variational loss using latent representations with mean and variance.
T ComputeVariationalLossBatch(Vector<T>[] studentMeans, Vector<T>[] studentLogVars, Vector<T>[] teacherMeans, Vector<T>[] teacherLogVars) Computes variational loss for batch of latent representations.
Vector<T> Reparameterize(Vector<T> mean, Vector<T> logVar, Vector<T> epsilon) Reparameterization trick for sampling from latent distribution during training.