ProbabilisticDistillationStrategy<T>
Models & Types ยท AiDotNet.KnowledgeDistillation.Strategies
Probabilistic distillation that transfers distributional knowledge by matching statistical properties.
Inheritance: DistillationStrategyBase<T> โ ProbabilisticDistillationStrategy<T>
How It Works
For Production Use: This strategy treats model outputs as samples from probability distributions and transfers knowledge about the entire distribution, not just point predictions. It matches statistical moments (mean, variance, higher moments) and can use measures like Maximum Mean Discrepancy (MMD).
Key Concept: Standard distillation matches individual predictions, but neural networks can be viewed as probabilistic models. This strategy captures uncertainty and distribution shape by matching:
- First moment (mean) - Expected predictions
- Second moment (variance) - Prediction uncertainty
- Distribution distance (MMD, Wasserstein) - Overall shape
Implementation: We provide three modes:
- MomentMatching: Match mean and variance of predictions across batch
- MaximumMeanDiscrepancy: Use MMD with RBF kernel to match distributions
- EntropyTransfer: Match prediction entropy (uncertainty calibration)
Research Basis: Based on probabilistic knowledge distillation and Bayesian neural networks. Particularly useful for uncertainty quantification and ensemble distillation.
Constructors
| Constructor | Summary |
|---|---|
ProbabilisticDistillationStrategy(double distributionWeight, ProbabilisticMode mode, double mmdBandwidth, double temperature, double alpha) |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
|
T ComputeDistributionalLoss(Vector<T>[] studentPredictions, Vector<T>[] teacherPredictions) |
Computes distributional loss by matching statistical properties across a batch. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |