KnowledgeDistillationTrainerBase<T, TInput, TOutput>
Base Classes · AiDotNet.KnowledgeDistillation
Abstract base class for all knowledge distillation trainers. Provides common functionality for training loops, data shuffling, validation, and evaluation.
Implements: IKnowledgeDistillationTrainer<T, TInput, TOutput>
Known derived types: KnowledgeDistillationTrainer<T>, SelfDistillationTrainer<T>
For Beginners
This base class implements the common training workflow shared by all distillation trainers. Specific trainer types (standard, self-distillation, online, etc.) inherit from this and customize only the parts that differ.
How It Works
Design Pattern: Template Method Pattern - the base class defines the training algorithm structure, and derived classes fill in specific steps.
Common Functionality Provided:
- Data shuffling using Fisher-Yates algorithm (O(n) efficiency)
- Epoch and batch management
- Validation after each epoch
- Progress callbacks
- Evaluation metrics (accuracy, loss)
- Teacher and strategy property management
Derived Classes Override:
- GetTeacherPredictions(): How to obtain teacher outputs for training
- OnEpochStart(): Custom logic before each epoch
- OnEpochEnd(): Custom logic after each epoch
- OnTrainingStart(): Custom logic before training begins
- OnTrainingEnd(): Custom logic after training completes
Properties
| Property | Summary |
|---|---|
IDistillationStrategy<T> DistillationStrategy { get; set; } |
Gets the distillation strategy for computing loss and gradients. |
ITeacherModel<TInput, TOutput> Teacher { get; set; } |
Gets the teacher model used for distillation. |
Methods
| Method | Summary |
|---|---|
T TrainBatch(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> inputs, Vector<TOutput> trueLabels) |
Trains the student model on a single batch using knowledge distillation. |
double Evaluate(Func<TInput, TOutput> studentForward, Vector<TInput> inputs, Vector<TOutput> trueLabels) |
Evaluates the student model’s accuracy on a dataset. |
void Train(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> trainInputs, Vector<TOutput> trainLabels, int epochs, int batchSize, Action<int, T> onEpochComplete) |
Trains the student model for multiple epochs (interface-compliant overload). |
void Train(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> trainInputs, Vector<TOutput> trainLabels, int epochs, int batchSize, Vector<TInput> validationInputs, Vector<TOutput> validationLabels, ICheckpointableModel student, Action<int, T> onEpochComplete) |
Trains the student model for multiple epochs using knowledge distillation. |