LAMBOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the LAMB (Layer-wise Adaptive Moments for Batch training) optimization algorithm.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → LAMBOptimizerOptions<T, TInput, TOutput>
For Beginners
LAMB is designed for training large models (like BERT, transformers) with very large batch sizes. It combines:
- From Adam: Adaptive learning rates that adjust per-parameter based on gradient history
- From LARS: Layer-wise scaling that stabilizes large batch training
The result is an optimizer that can train at batch sizes of 16K-32K while achieving the same accuracy as training with small batches, just much faster.
How It Works
LAMB combines Adam’s adaptive learning rates (first and second moment estimates) with LARS’s layer-wise trust ratio scaling. This enables training with extremely large batch sizes (up to 32K) while maintaining training stability and accuracy.
Based on the paper “Large Batch Optimization for Deep Learning: Training BERT in 76 minutes” by You et al. (2019).
Example
var batchSize = 32;
var options = new LAMBOptimizerOptions<float, Matrix<float>, Vector<float>>
{
InitialLearningRate = 0.00176 * Math.Sqrt(batchSize), // Square root scaling for LAMB
Beta1 = 0.9,
Beta2 = 0.999,
WeightDecay = 0.01,
BatchSize = 8192
};
var model = new SimpleRegression<float>();
var optimizer = new LAMBOptimizer<float, Matrix<float>, Vector<float>>(model, options);
Constructors
| Constructor | Summary |
|---|---|
LAMBOptimizerOptions() |
Properties
| Property | Summary |
|---|---|
bool ClipTrustRatio { get; set; } |
Gets or sets whether to clip the trust ratio to prevent extreme scaling. |
bool ExcludeBiasFromWeightDecay { get; set; } |
Gets or sets whether to exclude bias and normalization parameters from weight decay. |
bool UseBiasCorrection { get; set; } |
Gets or sets whether to use bias correction for the moment estimates. |
double Beta1 { get; set; } |
Gets or sets the exponential decay rate for the first moment estimates (momentum). |
double Beta2 { get; set; } |
Gets or sets the exponential decay rate for the second moment estimates. |
double Epsilon { get; set; } |
Gets or sets a small constant added to denominators to prevent division by zero. |
double InitialLearningRate { get; set; } |
Gets or sets the base learning rate for the LAMB optimizer. |
double MaxTrustRatio { get; set; } |
Gets or sets the maximum trust ratio when clipping is enabled. |
double WeightDecay { get; set; } |
Gets or sets the weight decay coefficient. |
int BatchSize { get; set; } |
Gets or sets the batch size for mini-batch gradient descent. |
int WarmupEpochs { get; set; } |
Gets or sets the number of warmup epochs for learning rate warmup. |
int[] LayerBoundaries { get; set; } |
Gets or sets the layer size boundaries for layer-wise scaling. |
int[] SkipTrustRatioLayers { get; set; } |
Gets or sets which layers should skip trust ratio scaling and use only Adam updates. |