LARSOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

Implements the LARS (Layer-wise Adaptive Rate Scaling) optimization algorithm.

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → LARSOptimizer<T, TInput, TOutput>

For Beginners

When training with very large batches (common in self-supervised learning like SimCLR), regular optimizers can become unstable because gradients get averaged over more samples, making them smaller. LARS solves this by looking at each layer and asking “how big are the weights compared to the gradients?” and scaling the learning rate accordingly. This allows stable training with batch sizes of 4096 or even larger.

How It Works

LARS is specifically designed for training with very large batch sizes (4096-32768). It automatically adapts the learning rate for each layer based on the ratio of parameter norm to gradient norm, which helps maintain stable training at scale.

Key Formula:

Based on the paper “Large Batch Training of Convolutional Networks” by You et al. (2017).

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new LARSOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with LARSOptimizer.");

Constructors

Constructor Summary
LARSOptimizer(IFullModel<T, TInput, TOutput> model, LARSOptimizerOptions<T, TInput, TOutput> options) Initializes a new instance of the LARSOptimizer class.

Properties

Property Summary
bool SupportsGpuUpdate { get; } Gets whether this optimizer supports GPU-accelerated parameter updates.
double Momentum { get; } Gets the current momentum coefficient.
double TrustCoefficient { get; } Gets the LARS trust coefficient.
double WeightDecay { get; } Gets the current weight decay coefficient.

Methods

Method Summary
LARSOptimizer<T, TInput, TOutput> CreateForFunction(LARSOptimizerOptions<T, TInput, TOutput> options) Creates a LARS optimizer for minimizing a plain function, with no model attached.
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) Updates a matrix of parameters using the LARS optimization algorithm.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() Gets the current optimizer options.
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) Performs the optimization process using the LARS algorithm.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses a LARS gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a matrix of parameters using the LARS optimization algorithm.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) Initializes LARS optimizer state on the GPU.
void Reset() Resets the optimizer’s internal state.
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) Updates parameters on the GPU using the LARS kernel.