AWGIMOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.MetaLearning.Options

Configuration for AWGIM, Attentive Weights Generation for few-shot learning via Information Maximization (Guo & Cheung, CVPR 2020).

Inheritance: ModelOptions → AWGIMOptions<T, TInput, TOutput>

Implements: IMetaLearnerOptions<T>

For Beginners

AWGIM writes a classifier for every query example on the spot. It looks at the whole support set to understand the task, looks at how the query relates to each support example, and turns both into the weights that classify that query. Two extra losses make the written weights carry as much information as possible about the task and the query.

How It Works

The defaults are the authors’ (github.com/Yiluan/AWGIM, main.py): a 128-wide latent space, 4-head attention, a two-layer weight decoder, dropout 0.3, AdamW at 2e-4 with weight decay 1e-6 and a staircase decay of 0.2 every 15,000 steps, gradients clipped at 0.1 by value and by norm, and loss weights α1 = 1 (support classification), α2 = α3 = 0.001 (the two reconstruction terms).

Constructors

Constructor Summary
AWGIMOptions(IFullModel<T, TInput, TOutput> metaModel) Initializes a new instance with the feature encoder whose embeddings AWGIM classifies.

Properties

Property Summary
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } Gets or sets the episodic data loader.
IFullModel<T, TInput, TOutput> MetaModel { get; set; } Gets or sets the feature encoder, which maps each example to one embedding row.
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } Not used: AWGIM generates weights by attention and has no inner optimization loop.
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } Not used: AWGIM runs its own AdamW meta-update (see OuterLearningRate).
ILossFunction<T> LossFunction { get; set; } Loss function for meta-training; null uses cross-entropy over logits.
bool EnableCheckpointing { get; set; } Gets or sets whether checkpointing is enabled.
bool UseFirstOrder { get; set; } Not applicable: there is no inner loop to differentiate through.
double ContextReconstructionWeight { get; set; } Gets or sets α2, the weight of reconstructing the contextual code from the generated weights.
double DropoutRate { get; set; } Gets or sets the dropout rate applied to MLP inputs and to the features being classified.
double InnerLearningRate { get; set; } Not an AWGIM hyperparameter: the method has no inner gradient loop.
double LearningRateDecayRate { get; set; } Gets or sets the staircase learning-rate decay factor.
double OuterLearningRate { get; set; } Gets or sets the AdamW learning rate of the meta-update.
double QueryReconstructionWeight { get; set; } Gets or sets α3, the weight of reconstructing the attentive (query) code from the generated weights.
double SupportClassificationWeight { get; set; } Gets or sets α1, the weight of classifying the support set with each query’s weights.
double WeightDecay { get; set; } Gets or sets the decoupled (AdamW) weight decay.
double? GradientClipThreshold { get; set; } Gets or sets the per-element gradient clipping threshold (clip by value).
double? GradientNormClipThreshold { get; set; } Gets or sets the gradient-norm clipping threshold, applied after value clipping.
int AdaptationSteps { get; set; } Not an AWGIM hyperparameter: weights are generated in one attention pass.
int CheckpointFrequency { get; set; } Gets or sets the checkpoint frequency.
int DecoderLayers { get; set; } Gets or sets the number of layers in the weight decoder and the reconstruction networks.
int EmbeddingDimension { get; set; } Gets or sets the width of the feature encoder’s per-example embedding (p).
int EvaluationFrequency { get; set; } Gets or sets how often to evaluate, in meta-iterations.
int EvaluationTasks { get; set; } Gets or sets the number of evaluation tasks.
int LatentDimension { get; set; } Gets or sets the latent width (d) of the encoders and the attention paths.
int LearningRateDecaySteps { get; set; } Gets or sets the number of meta-updates between learning-rate decays.
int MetaBatchSize { get; set; } Gets or sets the number of tasks per meta-update.
int NumClasses { get; set; } Gets or sets the number of classes (ways) per task.
int NumHeads { get; set; } Gets or sets the number of attention heads in every attention block.
int NumMetaIterations { get; set; } Gets or sets the number of meta-training iterations.
int? RandomSeed { get; set; } Gets or sets the random seed.

Methods

Method Summary
IMetaLearnerOptions<T> Clone() Creates a copy of these options that shares the feature encoder.
bool IsValid() Checks that the configuration describes a buildable AWGIM.