RAdamOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the RAdam (Rectified Adam) optimization algorithm, which adds a variance rectification term to Adam so that the adaptive learning rate is only trusted once enough gradient history exists to estimate it reliably.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → RAdamOptimizerOptions<T, TInput, TOutput>
For Beginners
Adam adapts its step size for every parameter by dividing by an estimate of how much each gradient has been bouncing around. Very early in training there are only a handful of gradients to estimate that from, so the estimate itself is unreliable, and dividing by an unreliable number can throw the model off course. The usual workaround is “warmup”: start with a tiny learning rate and ramp it up by hand. RAdam replaces that guesswork with a formula. It works out how trustworthy the estimate currently is, scales the step down accordingly, and — for the first handful of steps, when the estimate is not usable at all — skips the adaptive part entirely and takes a plain momentum step. You get Adam’s behaviour without having to tune a warmup schedule.
How It Works
RAdam was introduced in “On the Variance of the Adaptive Learning Rate and Beyond” (Liu et al., ICLR 2020, arXiv:1908.03265). The paper shows that Adam’s adaptive term has an undesirably large variance during the first few hundred steps — which is exactly why Adam so often needs a hand-tuned warmup schedule — and derives a closed-form rectification that removes the need for one.
Constructors
| Constructor | Summary |
|---|---|
RAdamOptimizerOptions() |
Initializes RAdam options with the paper-compatible defaults. |
RAdamOptimizerOptions(RAdamOptimizerOptions<T, TInput, TOutput> other) |
Creates a complete copy of an existing RAdam options instance. |
Properties
| Property | Summary |
|---|---|
double Beta1 { get; set; } |
Gets or sets the exponential decay rate for the first moment estimates (momentum). |
double Beta2 { get; set; } |
Gets or sets the exponential decay rate for the second moment estimates (adaptive learning rates). |
double Epsilon { get; set; } |
Gets or sets a small constant added to the denominator to improve numerical stability. |
double InitialLearningRate { get; set; } |
Gets or sets the initial step size used for parameter updates during optimization. |
double LearningRateDecreaseFactor { get; set; } |
Gets or sets the factor by which to decrease the learning rate when the loss is increasing or oscillating. |
double LearningRateIncreaseFactor { get; set; } |
Gets or sets the factor by which to increase the learning rate when the loss is consistently decreasing. |
double MaxLearningRate { get; set; } |
Gets or sets the maximum allowed learning rate during adaptive adjustments. |
double MinLearningRate { get; set; } |
Gets or sets the minimum allowed learning rate during adaptive adjustments. |
int BatchSize { get; set; } |
Gets or sets the batch size for mini-batch gradient descent. |