DAdaptAdamOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Options for D-Adaptation Adam (Defazio and Mishchenko 2023), which estimates the step size from the iterates instead of taking it from a tuned learning rate.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → DAdaptAdamOptimizerOptions<T, TInput, TOutput>

For Beginners

Instead of guessing how big each update should be, the optimizer measures how far the weights have travelled and grows its step to match, starting from a tiny one.

How It Works

The learning rate is a multiplier on the estimated distance d (1 by default, as the paper recommends), so a schedule such as a warmup and cosine decay still shapes the step. Defaults follow the reference implementation (facebookresearch/dadaptation DAdaptAdam, also vendored in audiocraft).

Constructors

Constructor Summary
DAdaptAdamOptimizerOptions()

Properties

Property Summary
bool DecoupleWeightDecay { get; set; } Gets or sets whether weight decay is decoupled, AdamW-style (true), rather than added to the gradient.
double Beta1 { get; set; } Gets or sets the first-moment decay β₁ (0.9).
double Beta2 { get; set; } Gets or sets the second-moment decay β₂ (0.999).
double Epsilon { get; set; } Gets or sets the denominator’s ε (1e-8).
double GrowthRate { get; set; } Gets or sets the most d may grow per step, as a factor (unbounded).
double InitialDistance { get; set; } Gets or sets the initial distance estimate d₀ (1e-6).
double InitialLearningRate { get; set; } Gets or sets the learning-rate multiplier on the estimated step size d (1).
double WeightDecay { get; set; } Gets or sets the weight decay (0).
int BatchSize { get; set; } Gets or sets the mini-batch size of Optimize (32).