LionOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Configuration options for the Lion (Evolved Sign Momentum) optimization algorithm.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → LionOptimizerOptions<T, TInput, TOutput>

For Beginners

Think of Lion as a streamlined version of Adam that focuses on the direction of learning (not the magnitude). It’s like a compass that only tells you which way to go, making decisions faster and using less memory. Lion is particularly effective for training large neural networks and transformers, where it can achieve better results than Adam while using half the memory.

How It Works

Lion is a modern optimization algorithm discovered through symbolic program search that offers significant advantages over Adam, including 50% memory reduction and superior performance on large models. Unlike Adam which maintains both momentum and variance, Lion uses only a single momentum state and relies on sign-based updates for improved efficiency and generalization.

Constructors

Constructor Summary
LionOptimizerOptions()

Properties

Property Summary
bool UseAdaptiveBeta1 { get; set; } Gets or sets whether to automatically adjust Beta1 during training.
bool UseAdaptiveBeta2 { get; set; } Gets or sets whether to automatically adjust Beta2 during training.
double Beta1 { get; set; } Gets or sets the exponential decay rate for the momentum interpolation (used for computing the update).
double Beta1DecreaseFactor { get; set; } Gets or sets the factor by which Beta1 is decreased when fitness does not improve.
double Beta1IncreaseFactor { get; set; } Gets or sets the factor by which Beta1 is increased when fitness improves.
double Beta2 { get; set; } Gets or sets the exponential decay rate for updating the momentum state.
double Beta2DecreaseFactor { get; set; } Gets or sets the factor by which Beta2 is decreased when fitness does not improve.
double Beta2IncreaseFactor { get; set; } Gets or sets the factor by which Beta2 is increased when fitness improves.
double InitialLearningRate { get; set; } Gets or sets the initial learning rate for the Lion optimizer.
double MaxBeta1 { get; set; } Gets or sets the maximum allowed value for Beta1.
double MaxBeta2 { get; set; } Gets or sets the maximum allowed value for Beta2.
double MinBeta1 { get; set; } Gets or sets the minimum allowed value for Beta1.
double MinBeta2 { get; set; } Gets or sets the minimum allowed value for Beta2.
double WeightDecay { get; set; } Gets or sets the weight decay (L2 regularization) coefficient.
int BatchSize { get; set; } Gets or sets the batch size for mini-batch gradient descent.