SparseMAMLOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.MetaLearning.Options

Configuration options for sparse-MAML, von Oswald et al., “Learning where to learn: Gradient sparsity in meta and continual learning” (NeurIPS 2021, arXiv:2110.14402).

Inheritance: ModelOptions → SparseMAMLOptions<T, TInput, TOutput>

Implements: IMetaLearnerOptions<T>

For Beginners

Learning from a handful of examples by changing every weight is a good way to memorize those examples and learn nothing general. These settings control a switch per weight saying whether it may change at all, and how fast those switches are tuned. They all start ON, so whichever ones end up off were turned off because that generalized better.

How It Works

sparse-MAML meta-learns WHICH weights the inner loop may change, rather than adapting all of them, so that “patterned sparsity emerges” and tasks interfere less.

Constructors

Constructor Summary
SparseMAMLOptions(IFullModel<T, TInput, TOutput> metaModel) Creates options for meta-learning metaModel, with the default settings.

Properties

Property Summary
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } Episodic data loader that supplies the meta-training tasks; null when tasks are passed to training directly.
IFullModel<T, TInput, TOutput> MetaModel { get; set; } The model whose parameters the meta-learner trains to adapt quickly to new tasks.
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } Must stay null: this algorithm takes no plain gradient inner step (it adapts without one, or by its own update rule), so a supplied optimizer is rejected when the learner is built.
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } Optimizer for the meta-update.
ILossFunction<T> LossFunction { get; set; } Loss function for meta-training; null uses the default multi-class classification loss.
SparseMAMLGateFunction GateFunction { get; set; } Gets or sets how a gate logit becomes the gate applied to the inner-loop update.
bool EnableCheckpointing { get; set; }
bool MetaLearnPerParameterRates { get; set; } Gets or sets whether per-parameter learning rates are meta-learned alongside the mask.
bool UseFirstOrder { get; set; }
double GateLearningRate { get; set; } Gets or sets the step size for meta-updating the gate logits.
double InitialGateLogit { get; set; } Gets or sets the initial value of every gate logit, which sets the starting sparsity.
double InnerLearningRate { get; set; }
double OuterLearningRate { get; set; }
double SparsityThreshold { get; set; } Gets or sets the gate value below which a weight counts as masked OUT when sparsity is reported.
double? GradientClipThreshold { get; set; }
int AdaptationSteps { get; set; }
int CheckpointFrequency { get; set; }
int EvaluationFrequency { get; set; }
int EvaluationTasks { get; set; }
int MetaBatchSize { get; set; }
int NumMetaIterations { get; set; }
int? RandomSeed { get; set; }

Methods

Method Summary
IMetaLearnerOptions<T> Clone()
bool IsValid()