HybridDistillationStrategy<T>

Models & Types ยท AiDotNet.KnowledgeDistillation.Strategies

Hybrid distillation strategy that combines multiple distillation strategies with configurable weights.

Inheritance: DistillationStrategyBase<T> โ†’ HybridDistillationStrategy<T>

How It Works

For Production Use: This strategy allows you to combine multiple distillation approaches (response-based, feature-based, attention-based, etc.) in a single training run. Each strategy contributes to the total loss based on its configured weight.

Example Use Case: For transformer distillation, combine:

  • 40% Response-based (output matching)
  • 30% Attention-based (attention pattern matching)
  • 30% Feature-based (intermediate layer matching)

This gives you comprehensive knowledge transfer at multiple levels.

Benefits:

  • Leverages multiple knowledge transfer mechanisms simultaneously
  • Weights can be tuned based on validation performance
  • More robust than single-strategy distillation
  • Commonly used in SOTA models like TinyBERT, MobileBERT

Constructors

Constructor Summary
HybridDistillationStrategy(ValueTuple<IDistillationStrategy<T>, double>[] strategies, double temperature, double alpha) Initializes a new instance of the HybridDistillationStrategy class.

Methods

Method Summary
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes combined gradient from all strategies.
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes combined loss from all strategies.
ValueTuple<IDistillationStrategy<T>, double>[] GetStrategies() Gets the individual strategies and their weights.