EnsembleTeacherModel<T>
Models & Types ยท AiDotNet.KnowledgeDistillation.Teachers
Ensemble teacher model that combines predictions from multiple teacher models.
Inheritance: TeacherModelBase<Vector<T>, Vector<T>, T> โ EnsembleTeacherModel<T>
For Beginners
Ensemble learning combines multiple models to create a stronger, more robust teacher. The intuition is similar to seeking advice from multiple experts rather than relying on a single expert.
How It Works
Benefits of Ensemble Teachers:
- Higher Accuracy: Ensemble outperforms individual models
- Better Calibration: Averaging reduces overconfidence
- Robustness: Less sensitive to individual model biases
- Knowledge Diversity: Student learns from complementary perspectives
Common Ensemble Strategies:
- Uniform Average: Equal weight to all teachers (default)
- Weighted Average: More weight to better-performing teachers
- Voting: For classification, majority vote
- Stacking: Meta-model combines predictions
Real-world Analogy: Imagine learning to play chess from multiple grandmasters. Each has different playing styles and strategies. By learning from all of them, you develop a more well-rounded understanding of the game than you would from just one teacher.
Practical Example: Train 3-5 models with different:
- Initializations (different random seeds)
- Architectures (CNN, ResNet, Transformer)
- Hyperparameters (learning rates, depths)
Combine them to create a powerful ensemble teacher.
References:
- You et al. (2017). Learning from Multiple Teacher Networks. KDD.
- Fukuda et al. (2017). Efficient Knowledge Distillation from an Ensemble of Teachers.
Constructors
| Constructor | Summary |
|---|---|
EnsembleTeacherModel(ITeacherModel<Vector<T>, Vector<T>>[] teachers, double[] weights, EnsembleAggregationMode aggregationMode) |
Initializes a new instance of the EnsembleTeacherModel class. |
Properties
| Property | Summary |
|---|---|
int OutputDimension { get; } |
Gets the output dimension (same for all teachers). |
int TeacherCount { get; } |
Gets the number of teachers in the ensemble. |
Methods
| Method | Summary |
|---|---|
Vector<T> GetLogits(Vector<T> input) |
Gets ensemble logits by combining predictions from all teachers. |
void UpdateWeights(Vector<T>[] validationInputs, Vector<T>[] validationLabels) |
Updates teacher weights based on performance (for adaptive weighting). |