DistillationLoss<T>

Models & Types · AiDotNet.KnowledgeDistillation

Implements the standard knowledge distillation loss function (Hinton et al., 2015). Combines hard loss (true labels) with soft loss (teacher predictions) using temperature scaling.

Inheritance: DistillationStrategyBase<T> → DistillationLoss<T>

For Beginners

This class calculates how well a student model is learning from a teacher model. It uses two types of information:

How It Works

  1. Soft Targets from the teacher: The teacher’s predictions, softened with temperature,

reveal relationships between classes (e.g., “dog” is more similar to “cat” than to “car”).

  1. Hard Targets (true labels): The actual correct answers for the training data.

The mathematics behind distillation loss:

Real-world Example: Imagine teaching a student to recognize animals. Instead of just saying “this is a dog” (hard label), you explain “this is definitely a dog (90% confident), it could be mistaken for a wolf (8%), but definitely not a cat (1%) or car (1%)”. This richer information helps the student learn better relationships between concepts.

References:

  • Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531

Constructors

Constructor Summary
DistillationLoss(double temperature, double alpha) Initializes a new instance of the DistillationLoss class.

Methods

Method Summary
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes the gradient of the distillation loss for backpropagation.
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes the combined distillation loss (soft loss from teacher + hard loss from true labels).