LarsGpuConfig

Options & Configuration · AiDotNet.Interfaces

Configuration for LARS (Layer-wise Adaptive Rate Scaling) optimizer on GPU.

Implements: IGpuOptimizerConfig

For Beginners

LARS was designed for training with huge batch sizes (like 32K images). It automatically adjusts the learning rate for each layer so that layers with large parameters don’t update too fast and layers with small parameters don’t update too slow.

How It Works

LARS scales the learning rate for each layer based on the ratio of parameter norm to gradient norm. This enables training with very large batch sizes.

Constructors

Constructor Summary
LarsGpuConfig(float learningRate, float momentum, float weightDecay, float trustCoefficient, int step) Creates a new LARS GPU configuration.

Properties

Property Summary
GpuOptimizerType OptimizerType { get; }
float LearningRate { get; set; }
float Momentum { get; set; } Gets the momentum coefficient (typically 0.9).
float TrustCoefficient { get; set; } Gets the trust coefficient for layer-wise scaling (typically 0.001).
float WeightDecay { get; set; }
int Step { get; set; }

Methods

Method Summary
void ApplyUpdate(IDirectGpuBackend backend, IGpuBuffer param, IGpuBuffer gradient, GpuOptimizerState state, int size)