MetaSGDAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

Implementation of Meta-SGD (Meta Stochastic Gradient Descent) algorithm.

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → MetaSGDAlgorithm<T, TInput, TOutput>

For Beginners

Meta-SGD learns how to update each parameter individually:

How It Works

Meta-SGD learns per-parameter learning rates for meta-learning. Instead of learning just initialization parameters like MAML, it learns the learning rate, momentum, and direction for each parameter individually, which can be seen as learning a custom optimizer for each parameter.

In regular training, you use one learning rate for all weights. But different parts of a neural network benefit from different learning rates. Meta-SGD figures this out automatically by learning:

  • α_i: The optimal learning rate for parameter i
  • β_i: The optimal momentum for parameter i (optional)
  • d_i: The optimal update direction/sign for parameter i (optional)

Algorithm - Meta-SGD:

Key Insights:

  1. Per-Parameter Optimization: Each parameter gets its own learned

optimizer configuration, allowing heterogeneous learning rates across layers.

  1. First-Order Method: No Hessian computation needed, much faster than

second-order MAML while maintaining strong performance.

  1. Interpretable: Learned per-parameter learning rates reveal which

parameters are most important for quick adaptation.

  1. Flexible Update Rules: Can combine with various base optimizers

(SGD, Adam, RMSprop) for different adaptation characteristics.

Reference: Li, Z., Zhou, F., Chen, F., & Li, H. (2017). Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.

Constructors

Constructor Summary
MetaSGDAlgorithm(MetaSGDOptions<T, TInput, TOutput> options) Initializes a new instance of the MetaSGDAlgorithm class.

Properties

Property Summary
MetaLearningAlgorithmType AlgorithmType { get; } Gets the algorithm type identifier for this meta-learner.

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) Adapts the meta-learned model to a new task using the learned per-parameter optimizers.
ModelOptions GetOptions()
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) Performs one meta-training step using Meta-SGD’s per-parameter optimization approach.