IDistillationStrategy<T>
Interfaces · AiDotNet.Interfaces
Defines a strategy for computing knowledge distillation loss between student and teacher models.
Known implementations: DistillationStrategyBase<T>
For Beginners
A distillation strategy determines how to measure the difference between what the student predicts and what the teacher predicts. Different strategies can focus on different aspects:
- Response-based: Compare final outputs
- Feature-based: Compare intermediate layer features
- Relation-based: Compare relationships between samples
How It Works
The most common approach (Hinton et al., 2015) combines two losses:
- Hard loss: How well the student matches the true labels
- Soft loss: How well the student mimics the teacher’s predictions
This combination allows the student to both get the right answers (hard loss) and learn the teacher’s reasoning (soft loss).
Batch Processing: This interface operates on batches (Matrix
Interface Design Note: This interface uses a single type parameter
Properties
| Property | Summary |
|---|---|
double Alpha { get; set; } |
Gets or sets the balance parameter (alpha) between hard loss and soft loss. |
double Temperature { get; set; } |
Gets or sets the temperature parameter for softening probability distributions. |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the gradient of the distillation loss for backpropagation. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the distillation loss between student and teacher batch outputs. |