LAMBOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

Implements the LAMB (Layer-wise Adaptive Moments for Batch training) optimization algorithm.

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → LAMBOptimizer<T, TInput, TOutput>

For Beginners

LAMB is the optimizer of choice for training large language models like BERT with massive batch sizes. It works by:

  • Computing Adam-style updates (momentum + adaptive learning rates)
  • Adding weight decay to prevent overfitting
  • Scaling the update per-layer based on weight/update magnitude ratios

This combination allows training to scale linearly with batch size while maintaining the same final accuracy as small-batch training.

How It Works

LAMB combines Adam’s adaptive learning rates with LARS’s layer-wise scaling, enabling training with extremely large batch sizes (up to 32K) while maintaining accuracy.

Key Formula:

Based on the paper “Large Batch Optimization for Deep Learning: Training BERT in 76 minutes” by You et al. (2019).

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new LAMBOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with LAMBOptimizer.");

Constructors

Constructor Summary
LAMBOptimizer(IFullModel<T, TInput, TOutput> model, LAMBOptimizerOptions<T, TInput, TOutput> options) Initializes a new instance of the LAMBOptimizer class.

Properties

Property Summary
bool SupportsGpuUpdate { get; } Gets whether this optimizer supports GPU-accelerated parameter updates.
double Beta1 { get; } Gets the current beta1 value.
double Beta2 { get; } Gets the current beta2 value.
double WeightDecay { get; } Gets the current weight decay coefficient.

Methods

Method Summary
LAMBOptimizer<T, TInput, TOutput> CreateForFunction(LAMBOptimizerOptions<T, TInput, TOutput> options) Creates a LAMB optimizer for minimizing a plain function, with no model attached.
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) Updates a matrix of parameters using the LAMB optimization algorithm.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() Gets the current optimizer options.
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) Performs the optimization process using the LAMB algorithm.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses a LAMB gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a matrix of parameters using the LAMB optimization algorithm.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) Initializes LAMB optimizer state on the GPU.
void Reset() Resets the optimizer’s internal state.
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) Updates parameters on the GPU using the LAMB kernel.