LEOAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

Implementation of Latent Embedding Optimization (LEO) meta-learning algorithm.

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → LEOAlgorithm<T, TInput, TOutput>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

LEO learns to write a small classifier for any new task: it summarises each class of the task in a short code, turns the codes into classifier weights, and nudges the codes (not the weights) until the classifier fits the few examples it was given.

How It Works

LEO (Rusu et al. 2019) generates the weights of a linear softmax classifier from a low-dimensional latent code and adapts the code rather than the weights. For each task:

Exact meta-gradients. Every inner step differentiates the support loss on a nested gradient tape with createGraph, so the step itself is part of the outer graph and the outer loop differentiates through it - the second-order terms included - into the encoder, relation network, decoder, learned step sizes and the feature encoder. UseFirstOrder stops the gradient at each inner gradient. This replaced forward differences over at most 500 weights per network, scaled up as if the rest behaved the same, and a feature-encoder update that differentiated its raw output against the labels.

Details the paper leaves open follow DeepMind’s reference implementation: the relation network is three bias-free layers with ReLU between them, a Gaussian’s scale is max(exp(u) - (1 - o), 1e-10) with offset o = 0 for the latent codes and o = sqrt(2 / (d + N)) (Glorot’s standard deviation for an N-way classifier over d-wide embeddings) for the classifier weights, the KL term is the mean over sampled codes of log q(z) - log p(z), the encoder penalty is a mean squared error, the orthogonality penalty is the mean squared deviation of the decoder’s row correlations from the identity, dropout acts on the feature embeddings, and evaluation uses means instead of samples.

Reference: Rusu, A. A., Rao, D., Sygnowski, J., et al. (2019). Meta-Learning with Latent Embedding Optimization. ICLR 2019.

Constructors

Constructor Summary
LEOAlgorithm(LEOOptions<T, TInput, TOutput> options) Initializes a new instance of the LEOAlgorithm class.

Properties

Property Summary
MetaLearningAlgorithmType AlgorithmType { get; } Gets the algorithm type identifier for this meta-learner.

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) Adapts to a new task: encode the support set to latent means, take the latent steps and the fine-tuning steps, and keep the resulting classifier.
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) Performs one meta-training step: each task’s objective (eq.