ANILAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

Implementation of Almost No Inner Loop (ANIL) meta-learning algorithm.

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → ANILAlgorithm<T, TInput, TOutput>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

Think of a neural network as having two parts: a body that turns each example into a description (an embedding), and a head that turns a description into a score for each class. ANIL keeps the body fixed while it adapts to a new task and only retrains the small head; the body improves across tasks.

How It Works

ANIL (Raghu et al. 2020) is MAML with the inner loop removed from every layer but the head: each task adapts only the final linear classifier on its support set, while the body (the meta-model, which embeds each example) is updated only by the outer loop. The outer loop meta-learns the body and the head’s initialisation together from the query loss after adaptation.

Per example. The body embeds every row of the input into a FeatureDimension-wide representation, and the head scores every row: logits = h W^T + b, one row of NumClasses scores per example. The loss is the configured loss on those logits against one class index per example - cross-entropy by default, the paper’s classification loss.

Exact meta-gradient. “We do not remove second order terms in ANIL (unlike in first-order MAML); second order terms still persist through the derivative of the inner loop update for the head parameters” (Raghu et al. 2020, App. C.1). The head’s initialisation receives prod_k (I - alpha H_k) applied to the query gradient, and the body receives, besides its direct query gradient, the cross term through each inner step’s support gradient. Both second-order pieces are Hessian-vector products taken as central differences of gradients along the head direction, so they cost two gradient evaluations per inner step. UseFirstOrder drops them.

Reference: Raghu, A., Raghu, M., Bengio, S., & Vinyals, O. (2020). Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML.

Constructors

Constructor Summary
ANILAlgorithm(ANILOptions<T, TInput, TOutput> options) Initializes a new instance of the ANILAlgorithm class.

Properties

Property Summary
MetaLearningAlgorithmType AlgorithmType { get; } Gets the algorithm type identifier for this meta-learner.

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) Adapts the meta-learned model to a new task by only updating the classification head.
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) Performs one meta-training step: head-only adaptation per task, then one update of the body and the head’s initialisation from the query losses.