AdamWOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the AdamW optimization algorithm with decoupled weight decay.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → AdamWOptimizerOptions<T, TInput, TOutput>
For Beginners
AdamW is an improved version of Adam that handles weight decay (a technique to prevent overfitting) in a mathematically cleaner way. The difference might seem subtle, but AdamW consistently achieves better results than Adam with L2 regularization, especially when training large models like transformers. If you’re not sure which to use, AdamW is generally the better choice.
How It Works
AdamW (Adam with decoupled Weight decay) differs from Adam with L2 regularization. In Adam with L2, weight decay is applied to the gradient before the adaptive learning rate is computed. In AdamW, weight decay is applied directly to the weights after the Adam update, which has been shown to improve generalization.
Based on the paper “Decoupled Weight Decay Regularization” by Ilya Loshchilov and Frank Hutter.
Constructors
| Constructor | Summary |
|---|---|
AdamWOptimizerOptions() |
Default ctor — overrides the base EnableGradientClipping default from false to true with MaxGradientNorm at the canonical transformer-training value of 1.0. |
AdamWOptimizerOptions(AdamWOptimizerOptions<T, TInput, TOutput> other) |
Copy constructor — required by the Options golden pattern so Clone() faithfully preserves every property. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Tensor<T>> MaskParameters { get; set; } |
Gets or sets the parameter tensors whose flattened concatenation WeightDecayMask and LearningRateMask describe, or null to read the masks positionally. |
OptimizerMomentStorage MomentStorage { get; set; } |
Gets or sets how the first and second moments are stored. |
Vector<T> LearningRateMask { get; set; } |
Gets or sets a per-parameter multiplier of the learning rate, or null to step every parameter at the same rate. |
Vector<T> WeightDecayMask { get; set; } |
Gets or sets a per-parameter multiplier applied to the decay term, or null to decay every parameter equally. |
bool UseAMSGrad { get; set; } |
Gets or sets whether to apply AMSGrad variant for improved convergence guarantees. |
bool UseAdaptiveBetas { get; set; } |
Gets or sets whether to automatically adjust the Beta parameters during training. |
double Beta1 { get; set; } |
Gets or sets the exponential decay rate for the first moment estimates (momentum). |
double Beta2 { get; set; } |
Gets or sets the exponential decay rate for the second moment estimates (adaptive learning rate). |
double Epsilon { get; set; } |
Gets or sets a small constant added to denominators to prevent division by zero. |
double InitialLearningRate { get; set; } |
Gets or sets the initial learning rate for the AdamW optimizer. |
double MaxBeta1 { get; set; } |
Gets or sets the maximum allowed value for Beta1. |
double MaxBeta2 { get; set; } |
Gets or sets the maximum allowed value for Beta2. |
double MinBeta1 { get; set; } |
Gets or sets the minimum allowed value for Beta1. |
double MinBeta2 { get; set; } |
Gets or sets the minimum allowed value for Beta2. |
double WeightDecay { get; set; } |
Gets or sets the weight decay coefficient (L2 penalty). |
int BatchSize { get; set; } |
Gets or sets the batch size for mini-batch gradient descent. |
int Int8MinQuantizedLength { get; set; } |
Gets or sets the smallest tensor that gets 8-bit moments in Int8Blockwise; shorter tensors (biases, norms) keep 32-bit moments. |
int Int8MomentBlockSize { get; set; } |
Gets or sets the number of elements that share one scale in Int8Blockwise. |