MetaSGDOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.MetaLearning.Options
Configuration options for the Meta-SGD (Meta Stochastic Gradient Descent) algorithm.
Inheritance: ModelOptions → MetaSGDOptions<T, TInput, TOutput>
Implements: IMetaLearnerOptions<T>
For Beginners
Think of Meta-SGD as “learning how to learn” at the finest grain:
How It Works
Meta-SGD extends MAML by learning not just the model initialization but also per-parameter learning rates, momentum terms, and update directions. This effectively learns a custom optimizer configuration for each parameter, enabling highly specialized adaptation strategies.
In standard training, you pick one learning rate for all parameters. But different parts of a neural network might benefit from different learning rates. Meta-SGD figures this out automatically by learning:
- Per-parameter learning rates: Some weights need small updates, others larger
- Per-parameter momentum: Some weights benefit from momentum, others don’t
- Update directions: Sometimes the gradient direction should be flipped or scaled
Algorithm Overview:
Key Insights:
- Per-parameter optimization allows heterogeneous learning rates across layers
- First-order method: no Hessian computation needed, much faster than second-order MAML
- Learned optimizers reveal which parameters are important for quick adaptation
- Can combine with various base update rules (SGD, Adam, RMSprop)
Reference: Li, Z., Zhou, F., Chen, F., & Li, H. (2017). Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.
Constructors
| Constructor | Summary |
|---|---|
MetaSGDOptions(IFullModel<T, TInput, TOutput> metaModel) |
Initializes a new instance of the MetaSGDOptions class with the required meta-model. |
Properties
| Property | Summary |
|---|---|
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } |
Gets or sets the episodic data loader for sampling tasks. |
IFullModel<T, TInput, TOutput> MetaModel { get; set; } |
Gets or sets the meta-model to be trained. |
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } |
Must stay null: Meta-SGD’s inner update is its learned per-parameter rule, so a supplied optimizer is rejected when the learner is built. |
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } |
Must stay null: this algorithm’s meta-update is part of the method rather than a plain gradient step, so a supplied optimizer is rejected when the learner is built. |
ILossFunction<T> LossFunction { get; set; } |
Loss function for meta-training; null uses the meta-model’s default loss. |
MetaSGDLearningRateInitialization LearningRateInitialization { get; set; } |
Gets or sets the initialization strategy for per-parameter learning rates. |
MetaSGDLearningRateScheduleType LearningRateSchedule { get; set; } |
Gets or sets the learning rate schedule type. |
MetaSGDUpdateRuleType UpdateRuleType { get; set; } |
Gets or sets the update rule type for per-parameter optimization. |
bool EnableCheckpointing { get; set; } |
Gets or sets whether to save checkpoints during training. |
bool LearnAdamBetas { get; set; } |
Gets or sets whether to learn Adam beta parameters when using Adam update rule. |
bool LearnDirection { get; set; } |
Gets or sets whether to learn per-parameter update direction signs. |
bool LearnLearningRate { get; set; } |
Gets or sets whether to learn per-parameter learning rates. |
bool LearnMomentum { get; set; } |
Gets or sets whether to learn per-parameter momentum coefficients. |
bool UseFirstOrder { get; set; } |
Gets or sets whether to use first-order approximation. |
bool UseHessianFree { get; set; } |
Gets or sets whether to use Hessian-free approximation for meta-gradients. |
bool UseLayerWiseDecay { get; set; } |
Gets or sets whether to apply layer-wise learning rate decay. |
bool UseLearningRateSchedule { get; set; } |
Gets or sets whether to use a learning rate schedule for meta-learning rates. |
bool UseParameterGrouping { get; set; } |
Gets or sets whether to use parameter grouping. |
bool UseParameterSharing { get; set; } |
Gets or sets whether to use parameter sharing based on similarity. |
bool UseTrustRegion { get; set; } |
Gets or sets whether to use trust region for parameter updates. |
bool UseWarmStart { get; set; } |
Gets or sets whether to use warm-start initialization for the optimizer. |
double AdamBeta1Init { get; set; } |
Gets or sets the initial value for Adam beta1. |
double AdamBeta2Init { get; set; } |
Gets or sets the initial value for Adam beta2. |
double AdamEpsilonInit { get; set; } |
Gets or sets the epsilon value for Adam numerical stability. |
double InnerLearningRate { get; set; } |
Gets or sets the learning rate for the inner loop (task adaptation). |
double LayerDecayFactor { get; set; } |
Gets or sets the decay factor per layer depth. |
double LearningRateInitRange { get; set; } |
Gets or sets the initialization range for learning rates when using random initialization. |
double LearningRateL2Reg { get; set; } |
Gets or sets the L2 regularization coefficient for learned learning rates. |
double OuterLearningRate { get; set; } |
Gets or sets the learning rate for the outer loop (meta-optimization). |
double ParameterSharingThreshold { get; set; } |
Gets or sets the similarity threshold for parameter sharing. |
double TrustRegionRadius { get; set; } |
Gets or sets the trust region radius. |
double? GradientClipThreshold { get; set; } |
Gets or sets the maximum gradient norm for gradient clipping. |
double? MaxLearningRate { get; set; } |
Gets or sets the maximum allowed per-parameter learning rate. |
double? MinLearningRate { get; set; } |
Gets or sets the minimum allowed per-parameter learning rate. |
int AdaptationSteps { get; set; } |
Gets or sets the number of gradient steps to take during inner loop adaptation. |
int CheckpointFrequency { get; set; } |
Gets or sets how often to save checkpoints. |
int EvaluationFrequency { get; set; } |
Gets or sets how often to evaluate during meta-training. |
int EvaluationTasks { get; set; } |
Gets or sets the number of tasks to use for evaluation. |
int InnerSteps { get; set; } |
Gets or sets the number of inner steps during meta-training. |
int MetaBatchSize { get; set; } |
Gets or sets the number of tasks to sample per meta-training iteration. |
int NumCurvatureSamples { get; set; } |
Gets or sets the number of samples for curvature approximation. |
int NumMetaIterations { get; set; } |
Gets or sets the total number of meta-training iterations to perform. |
int NumParameterGroups { get; set; } |
Gets or sets the number of parameter groups. |
int ScheduleWarmupEpisodes { get; set; } |
Gets or sets the number of warmup episodes for learning rate schedule. |
int? RandomSeed { get; set; } |
Gets or sets the random seed for reproducibility. |
Methods
| Method | Summary |
|---|---|
IMetaLearnerOptions<T> Clone() |
Creates a deep copy of the Meta-SGD options. |
bool IsValid() |
Validates that all Meta-SGD configuration options are properly set. |
int GetEffectiveParameterGroups(int numModelParameters) |
Gets the effective number of parameter groups. |
int GetTotalMetaParameters(int numModelParameters) |
Gets the total number of learnable meta-parameters. |