MetaOptNetAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
Implementation of Meta-learning with Differentiable Convex Optimization (MetaOptNet) algorithm.
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → MetaOptNetAlgorithm<T, TInput, TOutput>
Implements: IGeneratedParameterRegistrar<T>
For Beginners
MetaOptNet learns a feature extractor that produces embeddings where simple classifiers work well. The convex solver finds the best simple classifier for the few labelled examples, and the feature extractor is updated so that classifier does better on the rest of the task.
How It Works
MetaOptNet (Lee et al. 2019) uses a regularized linear classifier as the base learner: the embedding network produces features, a convex solver fits a classifier to the support set, and the query loss of that classifier trains the embedding. The paper’s base learner is the Crammer and Singer multi-class SVM in its dual form (eq. 10), with ridge regression and logistic regression as the other convex choices.
The gradient through the solver. The paper differentiates the base learner by applying the implicit
function theorem to the solver’s KKT conditions (section 3.2), not by differentiating its iterations. Each
solver here therefore runs numerically, off the tape, and its solution enters the tape as
theta = theta* - Jinv F(theta*, Z): the KKT residual F is rebuilt from the embeddings with engine
ops, and Jinv - the inverse KKT Jacobian at the solution - is a constant. The value is the solution
(the residual vanishes there) and the derivative is exactly the implicit one, so the embedding network’s
meta-gradient runs through the base learner at the cost of one multiplication by a constant matrix.
What this replaced. “SVM” was ridge regression on +/-1 labels; the linear system was solved by Gauss-Seidel iteration; logistic regression took fixed 0.1-sized steps and called them Newton’s method; and the encoder’s meta-gradient was the configured loss of its raw output against the labels, which never involved the solver at all. Query logits also flattened the batch into one vector.
Reference: Lee, K., Maji, S., Ravichandran, A., & Soatto, S. (2019). Meta-Learning with Differentiable Convex Optimization. CVPR 2019.
Constructors
| Constructor | Summary |
|---|---|
MetaOptNetAlgorithm(MetaOptNetOptions<T, TInput, TOutput> options) |
Initializes a new instance of the MetaOptNetAlgorithm class. |
Properties
| Property | Summary |
|---|---|
MetaLearningAlgorithmType AlgorithmType { get; } |
Gets the algorithm type identifier for this meta-learner. |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
Adapts to a new task by solving the convex base learner on its support set. |
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |
Performs one meta-training step: each task’s query loss under its solved base learner, differentiated through the solver into the embedding network and the logit scale. |