AdafactorOptimizerOptions<T, TInput, TOutput>

Options & Configuration · AiDotNet.Models.Options

Settings for AdafactorOptimizer.

Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → AdafactorOptimizerOptions<T, TInput, TOutput>

For Beginners

Adam remembers two numbers per weight to adapt each one’s step size. For a 1000x1000 layer that is two million extra numbers. Adafactor keeps a summary per row and per column instead — two thousand — and reconstructs an approximation when it needs one. It gives up a little precision to train models that otherwise would not fit.

How It Works

Adafactor (Shazeer and Stern 2018) replaces Adam’s full second-moment matrix with two vectors — a row sum and a column sum — so its optimizer state is O(n+m) rather than O(nm) for an n by m weight matrix. On a large embedding or projection that is the difference between an optimizer that fits in memory and one that does not.

Constructors

Constructor Summary
AdafactorOptimizerOptions() Initializes a new instance of the AdafactorOptimizerOptions class with the paper’s defaults.

Properties

Property Summary
bool Factored { get; set; } Whether matrices keep the factored (row and column) second moment; false keeps a full one per element.
bool UseRelativeStepSize { get; set; } Whether to derive the step size from the step number rather than use a fixed rate.
bool? MultiplyByParameterScale { get; set; } Whether the step size is multiplied by the RMS of the parameters it updates (floored at ParameterScaleFloor); null does so exactly when UseRelativeStepSize is on.
double Beta1 { get; set; } First-moment decay.
double Beta2 { get; set; } Second-moment decay.
double Epsilon { get; set; } Added to the squared gradient before it is accumulated.
double InitialLearningRate { get; set; } The step size.
double ParameterScaleFloor { get; set; } Floor on the parameter RMS used to scale the relative step.
double UpdateClippingThreshold { get; set; } Norm at which an update is scaled back down.
double WeightDecay { get; set; } Decoupled weight decay, applied to the parameters rather than the gradient.
int BatchSize { get; set; } Samples per optimizer step.