LarsGpuConfig
Options & Configuration · AiDotNet.Interfaces
Configuration for LARS (Layer-wise Adaptive Rate Scaling) optimizer on GPU.
Implements: IGpuOptimizerConfig
For Beginners
LARS was designed for training with huge batch sizes (like 32K images). It automatically adjusts the learning rate for each layer so that layers with large parameters don’t update too fast and layers with small parameters don’t update too slow.
How It Works
LARS scales the learning rate for each layer based on the ratio of parameter norm to gradient norm. This enables training with very large batch sizes.
Constructors
| Constructor | Summary |
|---|---|
LarsGpuConfig(float learningRate, float momentum, float weightDecay, float trustCoefficient, int step) |
Creates a new LARS GPU configuration. |
Properties
| Property | Summary |
|---|---|
GpuOptimizerType OptimizerType { get; } |
|
float LearningRate { get; set; } |
|
float Momentum { get; set; } |
Gets the momentum coefficient (typically 0.9). |
float TrustCoefficient { get; set; } |
Gets the trust coefficient for layer-wise scaling (typically 0.001). |
float WeightDecay { get; set; } |
|
int Step { get; set; } |
Methods
| Method | Summary |
|---|---|
void ApplyUpdate(IDirectGpuBackend backend, IGpuBuffer param, IGpuBuffer gradient, GpuOptimizerState state, int size) |