LambGpuConfig

Options & Configuration · AiDotNet.Interfaces

Configuration for LAMB (Layer-wise Adaptive Moments) optimizer on GPU.

Implements: IGpuOptimizerConfig

For Beginners

LAMB is like a combination of Adam and LARS. It uses Adam’s moment estimates for adaptive learning AND applies layer-wise scaling like LARS. This enables training very large models (like BERT) with very large batch sizes efficiently.

How It Works

LAMB combines Adam’s adaptive learning with LARS’s layer-wise scaling. It was designed for training BERT and other large transformers with huge batches.

Constructors

Constructor Summary
LambGpuConfig(float learningRate, float beta1, float beta2, float epsilon, float weightDecay, int step) Creates a new LAMB GPU configuration.

Properties

Property Summary
GpuOptimizerType OptimizerType { get; }
float Beta1 { get; set; } Gets the exponential decay rate for the first moment estimates (typically 0.9).
float Beta2 { get; set; } Gets the exponential decay rate for the second moment estimates (typically 0.999).
float Epsilon { get; set; } Gets the small constant for numerical stability (typically 1e-6).
float LearningRate { get; set; }
float WeightDecay { get; set; }
int Step { get; set; }

Methods

Method Summary
void ApplyUpdate(IDirectGpuBackend backend, IGpuBuffer param, IGpuBuffer gradient, GpuOptimizerState state, int size)