MemoryAwareSynapses<T, TInput, TOutput>

Models & Types · AiDotNet.ContinualLearning.Strategies

Memory Aware Synapses (MAS) strategy for continual learning.

Inheritance: ContinualLearningStrategyBase<T, TInput, TOutput> → MemoryAwareSynapses<T, TInput, TOutput>

For Beginners

MAS estimates weight importance in an unsupervised way by measuring how sensitive the network output is to each parameter. This means it doesn’t need task labels to compute importance - just unlabeled data!

How It Works

Key Insight: If changing a parameter causes a large change in the network output, that parameter is important and should be protected. This is measured by the gradient of the output norm with respect to each parameter.

How it works:

The Math:

For each sample x_n:

  1. Forward pass: y = F(x_n)

  2. Compute output norm gradient: ∂||y||²/∂θ = 2y × (∂y/∂θ)

  3. Importance: Ω_i = (1/N) × Σ_n |∂||y||²/∂θ_i|

Advantages over EWC:

Reference: Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. “Memory Aware Synapses: Learning what (not) to forget” (2018). ECCV.

Constructors

Constructor Summary
MemoryAwareSynapses(ILossFunction<T> lossFunction, MASOptions<T> options) Initializes a new MAS strategy with custom options.
MemoryAwareSynapses(ILossFunction<T> lossFunction, T lambda) Initializes a new MAS strategy with custom options.

Properties

Property Summary
IReadOnlyList<Vector<T>> TaskImportanceHistory { get; } Gets the importance history for all tasks.
T Lambda { get; } Gets the regularization strength.
Vector<T> ConsolidatedImportance { get; } Gets the consolidated importance values.
Vector<T> OptimalParameters { get; } Gets the optimal parameters from the last completed task.
bool ModifiesArchitecture { get; }
bool RequiresMemoryBuffer { get; }
long MemoryUsageBytes { get; }
string Name { get; }

Methods

Method Summary
T ComputeRegularizationLoss(IFullModel<T, TInput, TOutput> model)
Vector<T> AdjustGradients(IFullModel<T, TInput, TOutput> model, Vector<T> gradients)
void FinalizeTask(IFullModel<T, TInput, TOutput> model)
void PrepareForTask(IFullModel<T, TInput, TOutput> model, IDataset<T, TInput, TOutput> taskData)
void Reset()