ScheduleFreeAdamWOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Settings for ScheduleFreeAdamWOptimizer.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → ScheduleFreeAdamWOptimizerOptions<T, TInput, TOutput>

For Beginners

Most training lowers the learning rate as it goes, which means you have to decide up front how many steps you will run — stop early and the rate never came down, run longer and it came down too soon. This optimizer averages its recent positions instead, so it behaves well whenever you choose to stop.

How It Works

Schedule-Free AdamW (Defazio et al. 2024) removes the learning-rate schedule rather than replacing it. Instead of annealing a rate towards the end of a run whose length must be known in advance, it maintains a running average of the iterates and evaluates that. The averaging does the work a decay schedule normally does, and nothing has to know how long training will last.

Constructors

Constructor Summary
ScheduleFreeAdamWOptimizerOptions()

Properties

Property Summary
double Beta2 { get; set; } Second-moment decay for the AdamW step applied to the fast iterate.
double Epsilon { get; set; } Numerical-stability epsilon.
double InitialLearningRate { get; set; }
double Interpolation { get; set; } Where gradients are evaluated between the fast iterate and the running average.
double WeightDecay { get; set; } Decoupled weight decay.
int BatchSize { get; set; } Samples per optimizer step.
int WarmupSteps { get; set; } Steps over which the rate ramps up before the schedule-free behaviour takes over.