MetaSGDOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.MetaLearning.Options

Configuration options for the Meta-SGD (Meta Stochastic Gradient Descent) algorithm.

Inheritance: ModelOptions → MetaSGDOptions<T, TInput, TOutput>

Implements: IMetaLearnerOptions<T>

For Beginners

Think of Meta-SGD as “learning how to learn” at the finest grain:

How It Works

Meta-SGD extends MAML by learning not just the model initialization but also per-parameter learning rates, momentum terms, and update directions. This effectively learns a custom optimizer configuration for each parameter, enabling highly specialized adaptation strategies.

In standard training, you pick one learning rate for all parameters. But different parts of a neural network might benefit from different learning rates. Meta-SGD figures this out automatically by learning:

  • Per-parameter learning rates: Some weights need small updates, others larger
  • Per-parameter momentum: Some weights benefit from momentum, others don’t
  • Update directions: Sometimes the gradient direction should be flipped or scaled

Algorithm Overview:

Key Insights:

  1. Per-parameter optimization allows heterogeneous learning rates across layers
  2. First-order method: no Hessian computation needed, much faster than second-order MAML
  3. Learned optimizers reveal which parameters are important for quick adaptation
  4. Can combine with various base update rules (SGD, Adam, RMSprop)

Reference: Li, Z., Zhou, F., Chen, F., & Li, H. (2017). Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.

Constructors

Constructor Summary
MetaSGDOptions(IFullModel<T, TInput, TOutput> metaModel) Initializes a new instance of the MetaSGDOptions class with the required meta-model.

Properties

Property Summary
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } Gets or sets the episodic data loader for sampling tasks.
IFullModel<T, TInput, TOutput> MetaModel { get; set; } Gets or sets the meta-model to be trained.
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } Must stay null: Meta-SGD’s inner update is its learned per-parameter rule, so a supplied optimizer is rejected when the learner is built.
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } Must stay null: this algorithm’s meta-update is part of the method rather than a plain gradient step, so a supplied optimizer is rejected when the learner is built.
ILossFunction<T> LossFunction { get; set; } Loss function for meta-training; null uses the meta-model’s default loss.
MetaSGDLearningRateInitialization LearningRateInitialization { get; set; } Gets or sets the initialization strategy for per-parameter learning rates.
MetaSGDLearningRateScheduleType LearningRateSchedule { get; set; } Gets or sets the learning rate schedule type.
MetaSGDUpdateRuleType UpdateRuleType { get; set; } Gets or sets the update rule type for per-parameter optimization.
bool EnableCheckpointing { get; set; } Gets or sets whether to save checkpoints during training.
bool LearnAdamBetas { get; set; } Gets or sets whether to learn Adam beta parameters when using Adam update rule.
bool LearnDirection { get; set; } Gets or sets whether to learn per-parameter update direction signs.
bool LearnLearningRate { get; set; } Gets or sets whether to learn per-parameter learning rates.
bool LearnMomentum { get; set; } Gets or sets whether to learn per-parameter momentum coefficients.
bool UseFirstOrder { get; set; } Gets or sets whether to use first-order approximation.
bool UseHessianFree { get; set; } Gets or sets whether to use Hessian-free approximation for meta-gradients.
bool UseLayerWiseDecay { get; set; } Gets or sets whether to apply layer-wise learning rate decay.
bool UseLearningRateSchedule { get; set; } Gets or sets whether to use a learning rate schedule for meta-learning rates.
bool UseParameterGrouping { get; set; } Gets or sets whether to use parameter grouping.
bool UseParameterSharing { get; set; } Gets or sets whether to use parameter sharing based on similarity.
bool UseTrustRegion { get; set; } Gets or sets whether to use trust region for parameter updates.
bool UseWarmStart { get; set; } Gets or sets whether to use warm-start initialization for the optimizer.
double AdamBeta1Init { get; set; } Gets or sets the initial value for Adam beta1.
double AdamBeta2Init { get; set; } Gets or sets the initial value for Adam beta2.
double AdamEpsilonInit { get; set; } Gets or sets the epsilon value for Adam numerical stability.
double InnerLearningRate { get; set; } Gets or sets the learning rate for the inner loop (task adaptation).
double LayerDecayFactor { get; set; } Gets or sets the decay factor per layer depth.
double LearningRateInitRange { get; set; } Gets or sets the initialization range for learning rates when using random initialization.
double LearningRateL2Reg { get; set; } Gets or sets the L2 regularization coefficient for learned learning rates.
double OuterLearningRate { get; set; } Gets or sets the learning rate for the outer loop (meta-optimization).
double ParameterSharingThreshold { get; set; } Gets or sets the similarity threshold for parameter sharing.
double TrustRegionRadius { get; set; } Gets or sets the trust region radius.
double? GradientClipThreshold { get; set; } Gets or sets the maximum gradient norm for gradient clipping.
double? MaxLearningRate { get; set; } Gets or sets the maximum allowed per-parameter learning rate.
double? MinLearningRate { get; set; } Gets or sets the minimum allowed per-parameter learning rate.
int AdaptationSteps { get; set; } Gets or sets the number of gradient steps to take during inner loop adaptation.
int CheckpointFrequency { get; set; } Gets or sets how often to save checkpoints.
int EvaluationFrequency { get; set; } Gets or sets how often to evaluate during meta-training.
int EvaluationTasks { get; set; } Gets or sets the number of tasks to use for evaluation.
int InnerSteps { get; set; } Gets or sets the number of inner steps during meta-training.
int MetaBatchSize { get; set; } Gets or sets the number of tasks to sample per meta-training iteration.
int NumCurvatureSamples { get; set; } Gets or sets the number of samples for curvature approximation.
int NumMetaIterations { get; set; } Gets or sets the total number of meta-training iterations to perform.
int NumParameterGroups { get; set; } Gets or sets the number of parameter groups.
int ScheduleWarmupEpisodes { get; set; } Gets or sets the number of warmup episodes for learning rate schedule.
int? RandomSeed { get; set; } Gets or sets the random seed for reproducibility.

Methods

Method Summary
IMetaLearnerOptions<T> Clone() Creates a deep copy of the Meta-SGD options.
bool IsValid() Validates that all Meta-SGD configuration options are properly set.
int GetEffectiveParameterGroups(int numModelParameters) Gets the effective number of parameter groups.
int GetTotalMetaParameters(int numModelParameters) Gets the total number of learnable meta-parameters.