AutoLoRAOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.MetaLearning.Options

Configuration options for AutoLoRA (Zhang et al., NAACL 2024).

Inheritance: ModelOptions → AutoLoRAOptions<T, TInput, TOutput>

Implements: IMetaLearnerOptions<T>

How It Works

AutoLoRA automatically determines optimal per-layer LoRA ranks via meta-learning. Each rank-1 component has a continuous selection variable α ∈ [0,1] optimized on validation data (outer loop), while LoRA weights are trained on training data (inner loop). Final rank is determined by thresholding: k_l = |{α_{l,j} | α_{l,j} ≥ λ}|.

Constructors

Constructor Summary
AutoLoRAOptions(IFullModel<T, TInput, TOutput> metaModel) Creates options for meta-learning metaModel, with the default settings.

Properties

Property Summary
IEpisodicDataLoader<T, TInput, TOutput> DataLoader { get; set; } Episodic data loader that supplies the meta-training tasks; null when tasks are passed to training directly.
IFullModel<T, TInput, TOutput> MetaModel { get; set; } The model whose parameters the meta-learner trains to adapt quickly to new tasks.
IGradientBasedOptimizer<T, TInput, TOutput> InnerOptimizer { get; set; } Must stay null: this algorithm takes no plain gradient inner step (it adapts without one, or by its own update rule), so a supplied optimizer is rejected when the learner is built.
IGradientBasedOptimizer<T, TInput, TOutput> MetaOptimizer { get; set; } Optimizer for the meta-update.
ILossFunction<T> LossFunction { get; set; } Loss function for meta-training; null uses the default multi-class classification loss.
bool EnableCheckpointing { get; set; }
bool UseFirstOrder { get; set; } Whether to use the first-order approximation of the meta-gradient.
double InnerLearningRate { get; set; } Learning rate of the inner, per-task adaptation steps.
double OuterLearningRate { get; set; }
double RankRegularization { get; set; } Regularization penalty that encourages lower ranks (sparsity).
double RankThreshold { get; set; } Threshold for rank determination: α_{l,j} ≥ threshold means rank-1 component is kept.
double? GradientClipThreshold { get; set; }
int AdaptationSteps { get; set; }
int CheckpointFrequency { get; set; }
int EvaluationFrequency { get; set; }
int EvaluationTasks { get; set; }
int MaxRank { get; set; } Maximum rank per group (initial rank allocation).
int MetaBatchSize { get; set; }
int NumMetaIterations { get; set; }
int NumRankGroups { get; set; } Number of rank groups (analogous to layers).
int? RandomSeed { get; set; }

Methods

Method Summary
IMetaLearnerOptions<T> Clone()
bool IsValid()