LearningWithoutForgetting<T, TInput, TOutput>
Models & Types · AiDotNet.ContinualLearning.Strategies
Learning without Forgetting (LwF) strategy for continual learning.
Inheritance: ContinualLearningStrategyBase<T, TInput, TOutput> → LearningWithoutForgetting<T, TInput, TOutput>
Implements: IDistillationStrategy<T, TInput, TOutput>
For Beginners
LwF prevents forgetting by using knowledge distillation. Instead of storing old data, it stores the model’s predictions (knowledge) and trains the new model to match those predictions while also learning the new task.
How It Works
How it works:
The Math:
L_distill = T² * KL(softmax(z_teacher/T) || softmax(z_student/T))
Where T is the temperature and z are the logits (pre-softmax outputs).
Advantages:
Disadvantages:
Reference: Li and Hoiem “Learning without Forgetting” (ECCV 2016)
Constructors
| Constructor | Summary |
|---|---|
LearningWithoutForgetting(ILossFunction<T> lossFunction, LwFOptions<T> options) |
Initializes a new LwF strategy with custom options. |
LearningWithoutForgetting(ILossFunction<T> lossFunction, double distillationTemperature, double distillationWeight) |
Initializes a new LwF strategy with default options. |
Properties
| Property | Summary |
|---|---|
IFullModel<T, TInput, TOutput> TeacherModel { get; } |
|
ILossFunction<T> DistillationLoss { get; } |
The distillation loss as a differentiable objective: the configured loss between the current outputs (predicted) and the recorded outputs of the original network (target), times the effective distillation weight. |
T DistillationWeight { get; } |
|
T Temperature { get; } |
|
bool ModifiesArchitecture { get; } |
|
bool RequiresMemoryBuffer { get; } |
|
long MemoryUsageBytes { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
T ComputeDistillationLoss(TOutput teacherOutput, TOutput studentOutput) |
Computes the distillation loss between teacher and student predictions. |
T ComputeDistillationLoss(Vector<T> teacherOutput, Vector<T> studentOutput) |
Computes the distillation loss between teacher and student predictions. |
T ComputeRegularizationLoss(IFullModel<T, TInput, TOutput> model) |
|
ValueTuple<int, double> GetDistillationStats() |
Gets distillation statistics. |
Vector<T> AdjustGradients(IFullModel<T, TInput, TOutput> model, Vector<T> gradients) |
|
void AdvanceEpoch() |
Advances to the next epoch (for warmup tracking). |
void FinalizeTask(IFullModel<T, TInput, TOutput> model) |
|
void PrepareForTask(IFullModel<T, TInput, TOutput> model, IDataset<T, TInput, TOutput> taskData) |
|
void Reset() |