LARSOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the LARS (Layer-wise Adaptive Rate Scaling) optimization algorithm.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → LARSOptimizerOptions<T, TInput, TOutput>
For Beginners
When training with large batches (common in self-supervised learning), regular optimizers can become unstable. LARS solves this by automatically adjusting learning rates for each layer based on how “big” the weights and gradients are. This makes training more stable and allows you to use much larger batch sizes, which speeds up training significantly.
How It Works
LARS (Layer-wise Adaptive Rate Scaling) is designed for training with very large batch sizes (4096-32768). It automatically adapts the learning rate for each layer based on the ratio of parameter norm to gradient norm, which helps maintain stable training with large batches.
LARS is particularly important for self-supervised learning methods like SimCLR, which achieve their best results with batch sizes of 4096-8192.
Based on the paper “Large Batch Training of Convolutional Networks” by You et al. (2017).
Example
var options = new LARSOptimizerOptions<float, Matrix<float>, Vector<float>>
{
InitialLearningRate = 0.3, // Base learning rate (will be scaled per-layer)
Momentum = 0.9, // Standard momentum
WeightDecay = 1e-4, // Weight decay
TrustCoefficient = 0.001, // Controls layer-wise LR scaling
BatchSize = 4096 // Large batch size for SSL
};
var model = new SimpleRegression<float>();
var optimizer = new LARSOptimizer<float, Matrix<float>, Vector<float>>(model, options);
Constructors
| Constructor | Summary |
|---|---|
LARSOptimizerOptions() |
Properties
| Property | Summary |
|---|---|
bool ExcludeBiasFromLARS { get; set; } |
Gets or sets whether to exclude bias parameters and normalization layer parameters from LARS scaling. |
bool UseNesterov { get; set; } |
Gets or sets whether to use Nesterov momentum instead of standard momentum. |
double Epsilon { get; set; } |
Gets or sets a small constant added to denominators to prevent division by zero. |
double InitialLearningRate { get; set; } |
Gets or sets the base learning rate for the LARS optimizer. |
double Momentum { get; set; } |
Gets or sets the momentum coefficient for the optimizer. |
double TrustCoefficient { get; set; } |
Gets or sets the LARS trust coefficient (eta). |
double WeightDecay { get; set; } |
Gets or sets the weight decay coefficient. |
int BatchSize { get; set; } |
Gets or sets the batch size for mini-batch gradient descent. |
int WarmupEpochs { get; set; } |
Gets or sets the number of warmup steps for learning rate warmup. |
int[] LayerBoundaries { get; set; } |
Gets or sets the layer size boundaries for layer-wise scaling. |
int[] SkipLARSLayers { get; set; } |
Gets or sets which layers should skip LARS scaling and use only the base learning rate. |