ASGDOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements ASGD (Averaged Stochastic Gradient Descent): decayed SGD that additionally maintains a running average of the iterates, which is the quantity the method is named for and the one worth reading out.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → ASGDOptimizer<T, TInput, TOutput>
For Beginners
Ordinary SGD never really settles down – each step reacts to one noisy batch, so the model keeps jittering around the right answer instead of stopping on it. ASGD does not try to stop the jitter. It runs ordinary SGD and quietly keeps a running average of everywhere the model has been, because the average of many jittery positions is far closer to the truth than any one of them. It is the same reason the average of many dart throws finds the bullseye better than any single dart.
How It Works
The averaging result is Ruppert (1988) and Polyak and Juditsky, “Acceleration of Stochastic Approximation by Averaging” (SIAM J. Control Optim. 30(4):838-855, 1992, doi:10.1137/0330046): with a slowly decaying step size, the average of the iterates converges at the optimal asymptotic rate even though no individual iterate does. The schedule and decay parameterization used here follows Xu, “Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent” (arXiv:1107.2490, 2011). Per step t:
Note what mu_t does at the two ends of its range. While t is at or below t0 it equals 1, so ax simply
copies the current iterate and no averaging happens; past t0 it becomes 1/(t - t0), which is exactly the
incremental form of a running mean over the tail of the trajectory. This is why t0 defaults to 1e6: averaging
from step 1 would drag the answer back toward the initialization, so the average is deliberately started only
once the iterates are fluctuating around the solution rather than still travelling toward it.
eta_t is recomputed from gamma_0 and the step count on every step rather than being decayed in
place, so the schedule is exact and does not accumulate rounding drift over a long run. That also matches the
fused ASGD kernel, which is handed a freshly computed eta_t per step.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new ASGDOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with ASGDOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
ASGDOptimizer(IFullModel<T, TInput, TOutput> model, ASGDOptimizerOptions<T, TInput, TOutput> options) |
Initializes a new instance of the ASGDOptimizer class. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Retrieves the current options of the optimizer. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the ASGD algorithm. |
Vector<T> GetAveragedParameters() |
Gets the running average of the iterates, or null if no step has been taken yet. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses an ASGD gradient update to recover the parameters from before the step. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the ASGD optimization algorithm. |