SparseMAMLAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
sparse-MAML: meta-learn WHICH weights the inner loop is allowed to change, rather than adapting all of them.
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → SparseMAMLAlgorithm<T, TInput, TOutput>
Implements: IGeneratedParameterRegistrar<T>
For Beginners
When a model learns from only a handful of examples, changing every one of its millions of weights is a good way to memorize those examples and learn nothing general. This learns a switch for each weight saying whether it may change at all. The switches are tuned by what actually generalizes to held-out examples, and the useful pattern — most switches off — is something the training discovers rather than something imposed up front.
How It Works
von Oswald, Zhao, Kobayashi, Schug, Caccia, Zucchet and Sacramento, “Learning where to learn: Gradient sparsity in meta and continual learning” (NeurIPS 2021, arXiv:2110.14402). The premise: “this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn.” The reported consequence is that “patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis”, giving “better generalization and less interference in a range of few-shot and continual learning problems”.
Three properties carry the method, and all three are implemented rather than described.
Estimator. As in the paper, the step function is differentiated with the straight-through
estimator and second-order terms are dropped (first-order, like FOMAML), which gives Eq. 5. The
Sigmoid relaxation multiplies that by sigmoid’(m). Gate logits
start at the constant InitialGateLogit: a generic parameter vector carries no fan-in, so the
paper’s Kaiming initialization of m is not available.
Example
var architecture = new NeuralNetworkArchitecture<double>(inputFeatures: 4, outputSize: 2);
var model = new NeuralNetwork<double>(architecture);
var options = new SparseMAMLOptions<double, Tensor<double>, Tensor<double>>(model)
{
InitialGateLogit = 1.0, // start with every gate open; sparsity is learned, not imposed
GateLearningRate = 1e-2,
};
var sparseMaml = new SparseMAMLAlgorithm<double, Tensor<double>, Tensor<double>>(options);
var taskBatch = new TaskBatch<double, Tensor<double>, Tensor<double>>(
new IMetaLearningTask<double, Tensor<double>, Tensor<double>>[]
{
new MetaLearningTask<double, Tensor<double>, Tensor<double>>()
});
double metaLoss = Convert.ToDouble(sparseMaml.MetaTrain(taskBatch));
Constructors
| Constructor | Summary |
|---|---|
SparseMAMLAlgorithm(SparseMAMLOptions<T, TInput, TOutput> options) |
Initializes sparse-MAML over a model. |
Properties
| Property | Summary |
|---|---|
MetaLearningAlgorithmType AlgorithmType { get; } |
|
T Sparsity { get; } |
Gets the fraction of parameters currently gated OFF, i.e. |
Vector<T> GateLogits { get; } |
Gets the gate logits m. |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
|
T Gate(int index) |
Gets the gate for parameter index: 1[m >= 0] for the paper’s step gate, or sigmoid(m) for the relaxation. |
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |