MetaSGDAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
Implementation of Meta-SGD (Meta Stochastic Gradient Descent) algorithm.
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → MetaSGDAlgorithm<T, TInput, TOutput>
For Beginners
Meta-SGD learns how to update each parameter individually:
How It Works
Meta-SGD learns per-parameter learning rates for meta-learning. Instead of learning just initialization parameters like MAML, it learns the learning rate, momentum, and direction for each parameter individually, which can be seen as learning a custom optimizer for each parameter.
In regular training, you use one learning rate for all weights. But different parts of a neural network benefit from different learning rates. Meta-SGD figures this out automatically by learning:
- α_i: The optimal learning rate for parameter i
- β_i: The optimal momentum for parameter i (optional)
- d_i: The optimal update direction/sign for parameter i (optional)
Algorithm - Meta-SGD:
Key Insights:
- Per-Parameter Optimization: Each parameter gets its own learned
optimizer configuration, allowing heterogeneous learning rates across layers.
- First-Order Method: No Hessian computation needed, much faster than
second-order MAML while maintaining strong performance.
- Interpretable: Learned per-parameter learning rates reveal which
parameters are most important for quick adaptation.
- Flexible Update Rules: Can combine with various base optimizers
(SGD, Adam, RMSprop) for different adaptation characteristics.
Reference: Li, Z., Zhou, F., Chen, F., & Li, H. (2017). Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.
Constructors
| Constructor | Summary |
|---|---|
MetaSGDAlgorithm(MetaSGDOptions<T, TInput, TOutput> options) |
Initializes a new instance of the MetaSGDAlgorithm class. |
Properties
| Property | Summary |
|---|---|
MetaLearningAlgorithmType AlgorithmType { get; } |
Gets the algorithm type identifier for this meta-learner. |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
Adapts the meta-learned model to a new task using the learned per-parameter optimizers. |
ModelOptions GetOptions() |
|
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |
Performs one meta-training step using Meta-SGD’s per-parameter optimization approach. |