DistillationLoss<T>
Models & Types · AiDotNet.KnowledgeDistillation
Implements the standard knowledge distillation loss function (Hinton et al., 2015). Combines hard loss (true labels) with soft loss (teacher predictions) using temperature scaling.
Inheritance: DistillationStrategyBase<T> → DistillationLoss<T>
For Beginners
This class calculates how well a student model is learning from a teacher model. It uses two types of information:
How It Works
- Soft Targets from the teacher: The teacher’s predictions, softened with temperature,
reveal relationships between classes (e.g., “dog” is more similar to “cat” than to “car”).
- Hard Targets (true labels): The actual correct answers for the training data.
The mathematics behind distillation loss:
Real-world Example: Imagine teaching a student to recognize animals. Instead of just saying “this is a dog” (hard label), you explain “this is definitely a dog (90% confident), it could be mistaken for a wolf (8%), but definitely not a cat (1%) or car (1%)”. This richer information helps the student learn better relationships between concepts.
References:
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531
Constructors
| Constructor | Summary |
|---|---|
DistillationLoss(double temperature, double alpha) |
Initializes a new instance of the DistillationLoss class. |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the gradient of the distillation loss for backpropagation. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the combined distillation loss (soft loss from teacher + hard loss from true labels). |