ScheduleFreeAdamWOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Schedule-Free AdamW (Defazio et al. 2024): no learning-rate schedule, by averaging instead.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → ScheduleFreeAdamWOptimizer<T, TInput, TOutput>
For Beginners
Normal training slows down near the end so it can settle. That requires knowing when the end is. This keeps a running average of where it has been and reports that instead, so it is settled whenever you stop.
How It Works
Three sequences rather than one. z is the fast iterate that takes the AdamW step,
x is an equal-weighted running average of it, and gradients are evaluated at
y = (1 - beta) z + beta x, which sits between them. The averaging does the work that
annealing a learning rate normally does, so nothing needs to know the length of the run in
advance — which is the point: a scheduled run stopped early never finished decaying, and one
extended past its horizon decayed too soon.
From the paper, equations 3 to 5: y_t = (1 - beta) z_t + beta x_t, z_{t+1} = z_t - gamma g_t, and x_{t+1} = (1 - c_{t+1}) x_t + c_{t+1} z_{t+1} with c_{t+1} = 1/(t+1). Beta interpolates between Polyak-Ruppert averaging at 0 and primal averaging at 1.
The model’s parameters always hold y, never z or x. That is not an
implementation convenience: the gradient must be taken AT y for the method to be what the paper
describes, and the surrounding trainer computes gradients wherever the parameters currently are.
Reading the weights mid-run therefore gives y, which is also the point one should evaluate.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new ScheduleFreeAdamWOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with ScheduleFreeAdamWOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
ScheduleFreeAdamWOptimizer(IFullModel<T, TInput, TOutput> model, ScheduleFreeAdamWOptimizerOptions<T, TInput, TOutput> options) |
Creates a schedule-free AdamW optimizer for a model. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
Vector<T> AveragedParameters() |
The running average, which is the point the method converges on. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |