KnowledgeDistillationTrainerBase<T, TInput, TOutput>

Base Classes · AiDotNet.KnowledgeDistillation

Abstract base class for all knowledge distillation trainers. Provides common functionality for training loops, data shuffling, validation, and evaluation.

Implements: IKnowledgeDistillationTrainer<T, TInput, TOutput>

Known derived types: KnowledgeDistillationTrainer<T>, SelfDistillationTrainer<T>

For Beginners

This base class implements the common training workflow shared by all distillation trainers. Specific trainer types (standard, self-distillation, online, etc.) inherit from this and customize only the parts that differ.

How It Works

Design Pattern: Template Method Pattern - the base class defines the training algorithm structure, and derived classes fill in specific steps.

Common Functionality Provided:

  • Data shuffling using Fisher-Yates algorithm (O(n) efficiency)
  • Epoch and batch management
  • Validation after each epoch
  • Progress callbacks
  • Evaluation metrics (accuracy, loss)
  • Teacher and strategy property management

Derived Classes Override:

  • GetTeacherPredictions(): How to obtain teacher outputs for training
  • OnEpochStart(): Custom logic before each epoch
  • OnEpochEnd(): Custom logic after each epoch
  • OnTrainingStart(): Custom logic before training begins
  • OnTrainingEnd(): Custom logic after training completes

Properties

Property Summary
IDistillationStrategy<T> DistillationStrategy { get; set; } Gets the distillation strategy for computing loss and gradients.
ITeacherModel<TInput, TOutput> Teacher { get; set; } Gets the teacher model used for distillation.

Methods

Method Summary
T TrainBatch(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> inputs, Vector<TOutput> trueLabels) Trains the student model on a single batch using knowledge distillation.
double Evaluate(Func<TInput, TOutput> studentForward, Vector<TInput> inputs, Vector<TOutput> trueLabels) Evaluates the student model’s accuracy on a dataset.
void Train(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> trainInputs, Vector<TOutput> trainLabels, int epochs, int batchSize, Action<int, T> onEpochComplete) Trains the student model for multiple epochs (interface-compliant overload).
void Train(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> trainInputs, Vector<TOutput> trainLabels, int epochs, int batchSize, Vector<TInput> validationInputs, Vector<TOutput> validationLabels, ICheckpointableModel student, Action<int, T> onEpochComplete) Trains the student model for multiple epochs using knowledge distillation.