LEOOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.MetaLearning.Options

Configuration options for Latent Embedding Optimization (LEO) algorithm.

Inheritance: ModelOptions → LEOOptions<T, TInput, TOutput>

Implements: IMetaLearnerOptions<T>

For Beginners

Imagine you have a very large model with millions of parameters. Updating all of them during few-shot learning is slow and can lead to overfitting. LEO learns to “compress” these parameters into a much smaller space (like 64 numbers instead of millions). Adaptation happens in this compressed space, which is faster and more robust to overfitting.

How It Works

LEO performs meta-learning by learning a low-dimensional latent space for model parameters. Instead of adapting the full model parameters directly (like MAML), LEO:

Key Insight: Not all parameter configurations make sense for neural networks. By learning a latent space, LEO restricts adaptation to the “manifold” of sensible parameter settings, preventing bad updates.

Constructors

Constructor Summary
LEOOptions(IFullModel<T, TInput, TOutput> metaModel) Initializes a new instance of the LEOOptions class with the required meta-model.

Properties

Property Summary
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } Gets or sets the episodic data loader for sampling tasks.
IFullModel<T, TInput, TOutput> MetaModel { get; set; } Gets or sets the meta-model (feature encoder) to be trained.
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } Must stay null: this algorithm takes no plain gradient inner step (it adapts without one, or by its own update rule), so a supplied optimizer is rejected when the learner is built.
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } Optimizer for the meta-update.
ILossFunction<T> LossFunction { get; set; } Loss function for meta-training; null uses cross-entropy over logits.
bool EnableCheckpointing { get; set; } Gets or sets whether to save model checkpoints.
bool ShareEncoder { get; set; } Gets or sets whether to share the encoder across all classes.
bool UseFirstOrder { get; set; } Gets or sets whether to drop the second-order terms of the meta-gradient.
bool UseOrthogonalInit { get; set; } Gets or sets whether to initialise the decoder with orthonormal latent rows.
bool UseRelationEncoder { get; set; } Gets or sets whether to use a relation network for encoding.
double DropoutRate { get; set; } Gets or sets the dropout rate on the feature embeddings while meta-training.
double EncoderPenaltyWeight { get; set; } Gets or sets the weight of the penalty pulling the encoder’s initial codes toward the adapted codes.
double EntropyWeight { get; set; } Gets or sets the weight of an entropy bonus on the decoder’s weight distribution.
double FineTuningLearningRate { get; set; } Gets or sets the initial fine-tuning step size; LEO then learns one step size per embedding dimension.
double InnerLearningRate { get; set; } Gets or sets the initial latent step size; LEO then learns one step size per latent dimension.
double KLWeight { get; set; } Gets or sets the KL divergence weight for the latent space regularization.
double L2Regularization { get; set; } Gets or sets the L2 weight on the encoder, relation network and decoder.
double OrthogonalityWeight { get; set; } Gets or sets the weight of the decoder orthogonality penalty.
double OuterLearningRate { get; set; } Gets or sets the learning rate for the outer loop (meta-update).
double? GradientClipThreshold { get; set; } Gets or sets the maximum norm of each meta-gradient block.
int AdaptationSteps { get; set; } Gets or sets the number of adaptation steps in latent space.
int CheckpointFrequency { get; set; } Gets or sets how often to save checkpoints.
int EmbeddingDimension { get; set; } Gets or sets the width of each example’s feature embedding: the feature encoder’s per-example output width.
int EvaluationFrequency { get; set; } Gets or sets how often to evaluate the meta-learner.
int EvaluationTasks { get; set; } Gets or sets the number of tasks to use for evaluation.
int FineTuningSteps { get; set; } Gets or sets the number of fine-tuning steps taken directly on the generated classifier weights after the latent steps.
int HiddenDimension { get; set; } Gets or sets the width of the relation network’s hidden layers.
int LatentDimension { get; set; } Gets or sets the dimensionality of the latent space.
int MetaBatchSize { get; set; } Gets or sets the number of tasks to sample per meta-training iteration.
int NumClasses { get; set; } Gets or sets the number of output classes.
int NumMetaIterations { get; set; } Gets or sets the total number of meta-training iterations.
int? RandomSeed { get; set; } Gets or sets the random seed for reproducibility.

Methods

Method Summary
IMetaLearnerOptions<T> Clone() Creates a deep copy of the LEO options.
bool IsValid() Validates that all LEO configuration options are properly set.