IKnowledgeDistillationTrainer<T, TInput, TOutput>

Interfaces · AiDotNet.Interfaces

Defines the contract for knowledge distillation trainers that train student models using knowledge transferred from teacher models.

Known implementations: KnowledgeDistillationTrainerBase<T, TInput, TOutput>

For Beginners

A knowledge distillation trainer orchestrates the process of transferring knowledge from a large, accurate teacher model to a smaller, faster student model.

How It Works

Why an interface?

  • Flexibility: Multiple trainer implementations (standard, self-distillation, multi-teacher, etc.)
  • Testability: Easy to mock for unit testing
  • Extensibility: New training strategies can be added without breaking existing code
  • Dependency Injection: Can be injected into other components

Common Implementations:

  • Standard Trainer: Single teacher → single student
  • Self-Distillation Trainer: Model teaches itself (improves calibration)
  • Multi-Teacher Trainer: Multiple teachers → one student (ensemble distillation)
  • Online Trainer: Teacher updates during student training
  • Mutual Learning Trainer: Multiple students learn from each other

Properties

Property Summary
IDistillationStrategy<T> DistillationStrategy { get; } Gets the distillation strategy used for computing loss and gradients.
ITeacherModel<TInput, TOutput> Teacher { get; } Gets the teacher model used for distillation.

Methods

Method Summary
T TrainBatch(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> inputs, Vector<TOutput> trueLabels) Trains the student model on a single batch of data.
double Evaluate(Func<TInput, TOutput> studentForward, Vector<TInput> testInputs, Vector<TOutput> testLabels) Evaluates the student model’s accuracy on test data.
void Train(Func<TInput, TOutput> studentForward, Action<TOutput> studentBackward, Vector<TInput> trainInputs, Vector<TOutput> trainLabels, int epochs, int batchSize, Action<int, T> onEpochComplete) Trains the student model for multiple epochs.