ASGDOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Configuration options for ASGD (Averaged Stochastic Gradient Descent), which runs ordinary decayed SGD while maintaining a running average of the iterates and uses that average as the answer.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → ASGDOptimizerOptions<T, TInput, TOutput>

For Beginners

Plain SGD bounces around the answer rather than settling on it, because every step reacts to one noisy batch. ASGD does not try to stop the bouncing — it just keeps a running average of where the model has been, and the average lands much closer to the true answer than any individual position does. Think of estimating the centre of a dartboard: one dart tells you little, but the average of many darts is very accurate even if no single dart was good.

How It Works

The averaging idea is Ruppert (1988) and Polyak and Juditsky, “Acceleration of Stochastic Approximation by Averaging” (SIAM Journal on Control and Optimization 30(4), 1992, doi:10.1137/0330046): averaging the tail of an SGD trajectory attains the optimal asymptotic convergence rate even though each individual step does not.

The specific parameterization here — the learning-rate schedule gamma_t = gamma_0 (1 + agamma_0t)^-c and the decay term (1 - lambda*gamma_t) folded into the weight update — follows Xu, “Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent” (arXiv:1107.2490, 2011), which is also the form implemented by the fused ASGD kernel and by PyTorch’s torch.optim.ASGD.

Important: With the default T0 of 1,000,000 the averaging never actually starts in a normal-length run, and ASGD behaves as decayed SGD. That matches PyTorch’s default. To get the averaging benefit, set T0 to roughly the step at which you expect training to have settled — commonly halfway through the run.

Constructors

Constructor Summary
ASGDOptimizerOptions() Initializes ASGD options with the paper-compatible defaults.
ASGDOptimizerOptions(ASGDOptimizerOptions<T, TInput, TOutput> other) Creates a complete copy of an existing ASGD options instance.

Properties

Property Summary
double Alpha { get; set; } Gets or sets the decay exponent for the learning-rate schedule.
double InitialLearningRate { get; set; } Gets or sets the initial step size gamma_0 used for parameter updates.
double Lambda { get; set; } Gets or sets the decay coefficient lambda.
double LearningRateDecreaseFactor { get; set; } Gets or sets the factor by which to decrease the learning rate when the loss is increasing or oscillating.
double LearningRateIncreaseFactor { get; set; } Gets or sets the factor by which to increase the learning rate when the loss is consistently decreasing.
double MaxLearningRate { get; set; } Gets or sets the maximum allowed learning rate during adaptive adjustments.
double MinLearningRate { get; set; } Gets or sets the minimum allowed learning rate during adaptive adjustments.
double T0 { get; set; } Gets or sets the step at which averaging begins.
double WeightDecay { get; set; } Gets or sets the L2 weight decay added to the gradient.
int BatchSize { get; set; } Gets or sets the batch size for mini-batch gradient descent.