LambGpuConfig
Options & Configuration · AiDotNet.Interfaces
Configuration for LAMB (Layer-wise Adaptive Moments) optimizer on GPU.
Implements: IGpuOptimizerConfig
For Beginners
LAMB is like a combination of Adam and LARS. It uses Adam’s moment estimates for adaptive learning AND applies layer-wise scaling like LARS. This enables training very large models (like BERT) with very large batch sizes efficiently.
How It Works
LAMB combines Adam’s adaptive learning with LARS’s layer-wise scaling. It was designed for training BERT and other large transformers with huge batches.
Constructors
| Constructor | Summary |
|---|---|
LambGpuConfig(float learningRate, float beta1, float beta2, float epsilon, float weightDecay, int step) |
Creates a new LAMB GPU configuration. |
Properties
| Property | Summary |
|---|---|
GpuOptimizerType OptimizerType { get; } |
|
float Beta1 { get; set; } |
Gets the exponential decay rate for the first moment estimates (typically 0.9). |
float Beta2 { get; set; } |
Gets the exponential decay rate for the second moment estimates (typically 0.999). |
float Epsilon { get; set; } |
Gets the small constant for numerical stability (typically 1e-6). |
float LearningRate { get; set; } |
|
float WeightDecay { get; set; } |
|
int Step { get; set; } |
Methods
| Method | Summary |
|---|---|
void ApplyUpdate(IDirectGpuBackend backend, IGpuBuffer param, IGpuBuffer gradient, GpuOptimizerState state, int size) |