LearningWithoutForgetting<T, TInput, TOutput>

Models & Types · AiDotNet.ContinualLearning.Strategies

Learning without Forgetting (LwF) strategy for continual learning.

Inheritance: ContinualLearningStrategyBase<T, TInput, TOutput> → LearningWithoutForgetting<T, TInput, TOutput>

Implements: IDistillationStrategy<T, TInput, TOutput>

For Beginners

LwF prevents forgetting by using knowledge distillation. Instead of storing old data, it stores the model’s predictions (knowledge) and trains the new model to match those predictions while also learning the new task.

How It Works

How it works:

The Math:

L_distill = T² * KL(softmax(z_teacher/T) || softmax(z_student/T))

Where T is the temperature and z are the logits (pre-softmax outputs).

Advantages:

Disadvantages:

Reference: Li and Hoiem “Learning without Forgetting” (ECCV 2016)

Constructors

Constructor Summary
LearningWithoutForgetting(ILossFunction<T> lossFunction, LwFOptions<T> options) Initializes a new LwF strategy with custom options.
LearningWithoutForgetting(ILossFunction<T> lossFunction, double distillationTemperature, double distillationWeight) Initializes a new LwF strategy with default options.

Properties

Property Summary
IFullModel<T, TInput, TOutput> TeacherModel { get; }
ILossFunction<T> DistillationLoss { get; } The distillation loss as a differentiable objective: the configured loss between the current outputs (predicted) and the recorded outputs of the original network (target), times the effective distillation weight.
T DistillationWeight { get; }
T Temperature { get; }
bool ModifiesArchitecture { get; }
bool RequiresMemoryBuffer { get; }
long MemoryUsageBytes { get; }
string Name { get; }

Methods

Method Summary
T ComputeDistillationLoss(TOutput teacherOutput, TOutput studentOutput) Computes the distillation loss between teacher and student predictions.
T ComputeDistillationLoss(Vector<T> teacherOutput, Vector<T> studentOutput) Computes the distillation loss between teacher and student predictions.
T ComputeRegularizationLoss(IFullModel<T, TInput, TOutput> model)
ValueTuple<int, double> GetDistillationStats() Gets distillation statistics.
Vector<T> AdjustGradients(IFullModel<T, TInput, TOutput> model, Vector<T> gradients)
void AdvanceEpoch() Advances to the next epoch (for warmup tracking).
void FinalizeTask(IFullModel<T, TInput, TOutput> model)
void PrepareForTask(IFullModel<T, TInput, TOutput> model, IDataset<T, TInput, TOutput> taskData)
void Reset()