OnlineTeacherModel<T>
Models & Types · AiDotNet.KnowledgeDistillation.Teachers
Online teacher model that updates its parameters during student training.
Inheritance: TeacherModelBase<Vector<T>, Vector<T>, T> → OnlineTeacherModel<T>
For Beginners
Unlike standard distillation where the teacher is frozen, online distillation allows the teacher to continue learning during student training. This is useful for:
- Continuous learning scenarios
- Evolving data distributions
- Co-training teacher and student simultaneously
How It Works
How It Works:
- Initialize teacher model (can be pre-trained or random)
- During student training, also update teacher with new data
- Teacher provides evolving knowledge to student
- Both models improve together
Real-world Analogy: Imagine a mentor and apprentice both continuing to learn as they work together. The mentor (teacher) doesn’t just transfer old knowledge - they also learn from new experiences and share those insights with the apprentice (student).
Use Cases:
- Streaming Data: New data arrives continuously
- Domain Adaptation: Distribution shifts over time
- Co-training: Teacher and student help each other
- Incremental Learning: Models must adapt to new classes/tasks
Update Strategies:
- EMA (Exponential Moving Average): Smooth updates, stable teacher
- Periodic Sync: Update teacher every N steps
- Gradient-based: Teacher trained with separate loss
- Momentum: Teacher follows student with momentum
Advantages:
- Adapts to changing data
- No need for pre-trained teacher
- Can improve teacher and student together
- Suitable for lifelong learning
Challenges:
- Risk of teacher forgetting/degrading
- Need careful update rate tuning
- More complex training dynamics
- Harder to debug
References:
- Zhang et al. (2018). Deep Mutual Learning. CVPR.
- Anil et al. (2018). Large Scale Distributed Neural Network Training through Online Distillation.
Constructors
| Constructor | Summary |
|---|---|
OnlineTeacherModel(Func<Vector<T>, Vector<T>> teacherForward, int inputDimension, int outputDimension, Action<Vector<T>, Vector<T>> teacherUpdate, OnlineUpdateMode updateMode, double updateRate, int updateFrequency) |
Initializes a new instance of the OnlineTeacherModel class using function delegates. |
Properties
| Property | Summary |
|---|---|
bool IsUpdating { get; set; } |
Gets or sets whether the teacher is currently updating. |
int OutputDimension { get; } |
Gets the output dimension of the teacher model. |
Methods
| Method | Summary |
|---|---|
Vector<T> GetLogits(Vector<T> input) |
Gets logits from the teacher model. |
void PauseUpdates() |
Pauses teacher updates (freezes teacher). |
void ResetCounter() |
Resets the update counter. |
void ResumeUpdates() |
Resumes teacher updates. |
void Update(Vector<T> input, Vector<T> targetOutput) |
Updates the teacher model with new data. |