DAdaptAdamOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Options for D-Adaptation Adam (Defazio and Mishchenko 2023), which estimates the step size from the iterates instead of taking it from a tuned learning rate.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → DAdaptAdamOptimizerOptions<T, TInput, TOutput>
For Beginners
Instead of guessing how big each update should be, the optimizer measures how far the weights have travelled and grows its step to match, starting from a tiny one.
How It Works
The learning rate is a multiplier on the estimated distance d (1 by default, as the paper recommends), so a
schedule such as a warmup and cosine decay still shapes the step. Defaults follow the reference implementation
(facebookresearch/dadaptation DAdaptAdam, also vendored in audiocraft).
Constructors
| Constructor | Summary |
|---|---|
DAdaptAdamOptimizerOptions() |
Properties
| Property | Summary |
|---|---|
bool DecoupleWeightDecay { get; set; } |
Gets or sets whether weight decay is decoupled, AdamW-style (true), rather than added to the gradient. |
double Beta1 { get; set; } |
Gets or sets the first-moment decay β₁ (0.9). |
double Beta2 { get; set; } |
Gets or sets the second-moment decay β₂ (0.999). |
double Epsilon { get; set; } |
Gets or sets the denominator’s ε (1e-8). |
double GrowthRate { get; set; } |
Gets or sets the most d may grow per step, as a factor (unbounded). |
double InitialDistance { get; set; } |
Gets or sets the initial distance estimate d₀ (1e-6). |
double InitialLearningRate { get; set; } |
Gets or sets the learning-rate multiplier on the estimated step size d (1). |
double WeightDecay { get; set; } |
Gets or sets the weight decay (0). |
int BatchSize { get; set; } |
Gets or sets the mini-batch size of Optimize (32). |