AdafactorOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Settings for AdafactorOptimizer.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → AdafactorOptimizerOptions<T, TInput, TOutput>
For Beginners
Adam remembers two numbers per weight to adapt each one’s step size. For a 1000x1000 layer that is two million extra numbers. Adafactor keeps a summary per row and per column instead — two thousand — and reconstructs an approximation when it needs one. It gives up a little precision to train models that otherwise would not fit.
How It Works
Adafactor (Shazeer and Stern 2018) replaces Adam’s full second-moment matrix with two vectors — a row sum and a column sum — so its optimizer state is O(n+m) rather than O(nm) for an n by m weight matrix. On a large embedding or projection that is the difference between an optimizer that fits in memory and one that does not.
Constructors
| Constructor | Summary |
|---|---|
AdafactorOptimizerOptions() |
Initializes a new instance of the AdafactorOptimizerOptions class with the paper’s defaults. |
Properties
| Property | Summary |
|---|---|
bool Factored { get; set; } |
Whether matrices keep the factored (row and column) second moment; false keeps a full one per element. |
bool UseRelativeStepSize { get; set; } |
Whether to derive the step size from the step number rather than use a fixed rate. |
bool? MultiplyByParameterScale { get; set; } |
Whether the step size is multiplied by the RMS of the parameters it updates (floored at ParameterScaleFloor); null does so exactly when UseRelativeStepSize is on. |
double Beta1 { get; set; } |
First-moment decay. |
double Beta2 { get; set; } |
Second-moment decay. |
double Epsilon { get; set; } |
Added to the squared gradient before it is accumulated. |
double InitialLearningRate { get; set; } |
The step size. |
double ParameterScaleFloor { get; set; } |
Floor on the parameter RMS used to scale the relative step. |
double UpdateClippingThreshold { get; set; } |
Norm at which an update is scaled back down. |
double WeightDecay { get; set; } |
Decoupled weight decay, applied to the parameters rather than the gradient. |
int BatchSize { get; set; } |
Samples per optimizer step. |