DistillationStrategyBase<T>
Base Classes · AiDotNet.KnowledgeDistillation
Abstract base class for knowledge distillation strategies. Provides common functionality for computing losses and gradients in student-teacher training.
Implements: IDistillationStrategy<T>
Known derived types: AdaptiveDistillationStrategyBase<T>, AttentionDistillationStrategy<T>, ContrastiveDistillationStrategy<T>, CurriculumDistillationStrategyBase<T>, DistillationLoss<T>, FactorTransferDistillationStrategy<T>, FlowBasedDistillationStrategy<T>, HybridDistillationStrategy<T>, NeuronSelectivityDistillationStrategy<T>, ProbabilisticDistillationStrategy<T>, RelationalDistillationStrategy<T>, SimilarityPreservingStrategy<T>, VariationalDistillationStrategy<T>
For Beginners
A distillation strategy defines how to measure the difference between student and teacher predictions. This base class provides common functionality that all strategies need, like temperature and alpha parameters.
How It Works
Design Philosophy: Different distillation strategies focus on different aspects:
- Response-based: Match final outputs (logits/probabilities)
- Feature-based: Match intermediate layer representations
- Relation-based: Match relationships between samples
- Attention-based: Match attention patterns (for transformers)
This base class ensures all strategies handle temperature and alpha consistently, while allowing flexibility in how loss is computed.
Batch Processing: All strategies now operate on batches (Matrix
Template Method Pattern: The base class defines the structure (properties, validation), and subclasses implement the specifics (loss computation logic).
Properties
| Property | Summary |
|---|---|
double Alpha { get; set; } |
Gets or sets the balance parameter between hard loss and soft loss. |
double Temperature { get; set; } |
Gets or sets the temperature parameter for softening probability distributions. |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the gradient of the distillation loss for backpropagation. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the distillation loss between student and teacher batch outputs. |