OptimizerKind

Enums · AiDotNet.Enums

Identifies the optimizer a research paper specifies for a model.

For Beginners

Research papers do not just pick a learning rate — they pick an optimizer too, and the two go together. A learning rate that is right for SGD is usually wrong for Adam. Recording which optimizer the paper used is what makes its other numbers meaningful.

How It Works

Used by PaperOptimizerAttribute so a model can declare which optimizer its paper trains with, alongside the hyperparameters that paper states.

An enum rather than a string so a typo cannot silently produce a model that matches no optimizer and quietly keeps the library default.

Fields

Field Summary
OptimizerKind Adadelta Adadelta.
OptimizerKind Adafactor Adafactor: Adam-style adaptivity with factored, O(n+m) second moments.
OptimizerKind Adagrad Adagrad.
OptimizerKind Adam Adam (Kingma and Ba, 2015).
OptimizerKind Adam8Bit Adam with 8-bit optimizer state.
OptimizerKind AdamW AdamW, Adam with decoupled weight decay (Loshchilov and Hutter, 2019).
OptimizerKind Adamax Adamax.
OptimizerKind DAdaptAdam D-Adaptation Adam: Adam whose step size is estimated from the iterates.
OptimizerKind LBfgs L-BFGS, limited-memory quasi-Newton.
OptimizerKind Lamb LAMB, layer-wise adaptive moments for large-batch training.
OptimizerKind Lion Lion (Chen et al., 2023).
OptimizerKind Nadam Nadam, Adam with Nesterov momentum.
OptimizerKind RAdam RAdam, Adam with a rectified adaptive learning rate (Liu et al.
OptimizerKind RmsProp RMSProp.
OptimizerKind ScaledAdam ScaledAdam: Adam scaled by each tensor’s rms, learning each tensor’s scale (k2-fsa icefall).
OptimizerKind ScheduleFreeAdamW Schedule-Free AdamW: no learning-rate schedule, by iterate averaging.
OptimizerKind Sgd Stochastic gradient descent.
OptimizerKind SgdMomentum SGD with momentum.
OptimizerKind Unspecified The paper does not state an optimizer, or it has not been recorded yet.
int value__