WarpGradAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

Implementation of WarpGrad (Meta-Learning with Warped Gradient Descent) for few-shot learning.

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → WarpGradAlgorithm<T, TInput, TOutput>

For Beginners

WarpGrad improves how gradient descent works, not just where it starts:

How it works:

  1. Standard gradient descent follows the steepest direction downhill
  2. But “steepest” depends on how you measure distance in parameter space
  3. WarpGrad learns warp-layers that transform gradients before they’re applied
  4. These transformed gradients lead to more effective parameter updates
  5. The warp-layers are shared across all tasks (meta-learned)

Analogy: Imagine navigating a city to reach your destination:

  • Standard gradient descent: Always walk directly toward the destination (may hit walls)
  • MAML: Start from a good location in the city
  • WarpGrad: Learn the city’s road network so you always take efficient routes

Why it’s better than MAML:

  • No second-order gradients needed (no backprop through inner loop)
  • Warp-layers provide task-independent preconditioning
  • Can use any number of inner-loop steps cheaply
  • Naturally handles different parameter scales across layers

How It Works

WarpGrad learns a preconditioning transformation (warp-layers) that reshapes the gradient descent landscape to make inner-loop adaptation more efficient. Unlike MAML which only learns a good initialization, WarpGrad learns a good optimization geometry.

Algorithm - WarpGrad:

Key Insights:

  1. Gradient Geometry: Warp-layers learn a Riemannian metric that makes gradient

descent more effective, similar to natural gradient methods but task-adapted.

  1. No Inner-Loop Backprop: Unlike MAML, gradients for warp-layers flow through

the warp transformation only, not through the entire inner-loop trajectory. This makes WarpGrad O(K) per task instead of MAML’s O(K^2).

  1. Complementary to Initialization: WarpGrad improves both WHERE you start (theta)

and HOW you move (W), providing two orthogonal axes of meta-learning.

  1. Identity Initialization: Warp-layers start near identity (no warping) and

gradually learn useful transformations during meta-training.

Reference: Flennerhag, S., Rusu, A. A., Pascanu, R., Visin, F., Yin, H., & Hadsell, R. (2020). Meta-Learning with Warped Gradient Descent. ICLR 2020.

Constructors

Constructor Summary
WarpGradAlgorithm(WarpGradOptions<T, TInput, TOutput> options) Initializes a new instance of the WarpGrad algorithm.

Properties

Property Summary
IReadOnlyList<Vector<T>> WarpLayerParameters { get; } Gets the warp-layer parameters for inspection or serialization.
MetaLearningAlgorithmType AlgorithmType { get; }

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) Adapts the meta-learned model to a new task using warped gradient descent.
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) Performs one meta-training step on a batch of tasks using warped gradient descent.