AdamWGpuConfig
Options & Configuration · AiDotNet.Interfaces
Configuration for AdamW optimizer on GPU.
Implements: IGpuOptimizerConfig
For Beginners
AdamW fixes a subtle issue with L2 regularization in Adam. The original Adam with weight decay doesn’t properly regularize because the adaptive learning rates interfere. AdamW applies weight decay directly to weights, which works better.
How It Works
AdamW is Adam with decoupled weight decay. Instead of adding weight decay to the gradient before the Adam update, it subtracts it directly from the weights after the update.
Constructors
| Constructor | Summary |
|---|---|
AdamWGpuConfig(float learningRate, float beta1, float beta2, float epsilon, float weightDecay, int step) |
Creates a new AdamW GPU configuration. |
Properties
| Property | Summary |
|---|---|
GpuOptimizerType OptimizerType { get; } |
|
float Beta1 { get; set; } |
Gets the exponential decay rate for the first moment estimates (typically 0.9). |
float Beta2 { get; set; } |
Gets the exponential decay rate for the second moment estimates (typically 0.999). |
float Epsilon { get; set; } |
Gets the small constant for numerical stability (typically 1e-8). |
float LearningRate { get; set; } |
|
float WeightDecay { get; set; } |
|
int Step { get; set; } |
Methods
| Method | Summary |
|---|---|
void ApplyUpdate(IDirectGpuBackend backend, IGpuBuffer param, IGpuBuffer gradient, GpuOptimizerState state, int size) |