WarpGradAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
Implementation of WarpGrad (Meta-Learning with Warped Gradient Descent) for few-shot learning.
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → WarpGradAlgorithm<T, TInput, TOutput>
For Beginners
WarpGrad improves how gradient descent works, not just where it starts:
How it works:
- Standard gradient descent follows the steepest direction downhill
- But “steepest” depends on how you measure distance in parameter space
- WarpGrad learns warp-layers that transform gradients before they’re applied
- These transformed gradients lead to more effective parameter updates
- The warp-layers are shared across all tasks (meta-learned)
Analogy: Imagine navigating a city to reach your destination:
- Standard gradient descent: Always walk directly toward the destination (may hit walls)
- MAML: Start from a good location in the city
- WarpGrad: Learn the city’s road network so you always take efficient routes
Why it’s better than MAML:
- No second-order gradients needed (no backprop through inner loop)
- Warp-layers provide task-independent preconditioning
- Can use any number of inner-loop steps cheaply
- Naturally handles different parameter scales across layers
How It Works
WarpGrad learns a preconditioning transformation (warp-layers) that reshapes the gradient descent landscape to make inner-loop adaptation more efficient. Unlike MAML which only learns a good initialization, WarpGrad learns a good optimization geometry.
Algorithm - WarpGrad:
Key Insights:
- Gradient Geometry: Warp-layers learn a Riemannian metric that makes gradient
descent more effective, similar to natural gradient methods but task-adapted.
- No Inner-Loop Backprop: Unlike MAML, gradients for warp-layers flow through
the warp transformation only, not through the entire inner-loop trajectory. This makes WarpGrad O(K) per task instead of MAML’s O(K^2).
- Complementary to Initialization: WarpGrad improves both WHERE you start (theta)
and HOW you move (W), providing two orthogonal axes of meta-learning.
- Identity Initialization: Warp-layers start near identity (no warping) and
gradually learn useful transformations during meta-training.
Reference: Flennerhag, S., Rusu, A. A., Pascanu, R., Visin, F., Yin, H., & Hadsell, R. (2020). Meta-Learning with Warped Gradient Descent. ICLR 2020.
Constructors
| Constructor | Summary |
|---|---|
WarpGradAlgorithm(WarpGradOptions<T, TInput, TOutput> options) |
Initializes a new instance of the WarpGrad algorithm. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Vector<T>> WarpLayerParameters { get; } |
Gets the warp-layer parameters for inspection or serialization. |
MetaLearningAlgorithmType AlgorithmType { get; } |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
Adapts the meta-learned model to a new task using warped gradient descent. |
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |
Performs one meta-training step on a batch of tasks using warped gradient descent. |