OptimizerKind
Enums · AiDotNet.Enums
Identifies the optimizer a research paper specifies for a model.
For Beginners
Research papers do not just pick a learning rate — they pick an optimizer too, and the two go together. A learning rate that is right for SGD is usually wrong for Adam. Recording which optimizer the paper used is what makes its other numbers meaningful.
How It Works
Used by PaperOptimizerAttribute so a model can declare which
optimizer its paper trains with, alongside the hyperparameters that paper states.
An enum rather than a string so a typo cannot silently produce a model that matches no optimizer and quietly keeps the library default.
Fields
| Field | Summary |
|---|---|
OptimizerKind Adadelta |
Adadelta. |
OptimizerKind Adafactor |
Adafactor: Adam-style adaptivity with factored, O(n+m) second moments. |
OptimizerKind Adagrad |
Adagrad. |
OptimizerKind Adam |
Adam (Kingma and Ba, 2015). |
OptimizerKind Adam8Bit |
Adam with 8-bit optimizer state. |
OptimizerKind AdamW |
AdamW, Adam with decoupled weight decay (Loshchilov and Hutter, 2019). |
OptimizerKind Adamax |
Adamax. |
OptimizerKind DAdaptAdam |
D-Adaptation Adam: Adam whose step size is estimated from the iterates. |
OptimizerKind LBfgs |
L-BFGS, limited-memory quasi-Newton. |
OptimizerKind Lamb |
LAMB, layer-wise adaptive moments for large-batch training. |
OptimizerKind Lion |
Lion (Chen et al., 2023). |
OptimizerKind Nadam |
Nadam, Adam with Nesterov momentum. |
OptimizerKind RAdam |
RAdam, Adam with a rectified adaptive learning rate (Liu et al. |
OptimizerKind RmsProp |
RMSProp. |
OptimizerKind ScaledAdam |
ScaledAdam: Adam scaled by each tensor’s rms, learning each tensor’s scale (k2-fsa icefall). |
OptimizerKind ScheduleFreeAdamW |
Schedule-Free AdamW: no learning-rate schedule, by iterate averaging. |
OptimizerKind Sgd |
Stochastic gradient descent. |
OptimizerKind SgdMomentum |
SGD with momentum. |
OptimizerKind Unspecified |
The paper does not state an optimizer, or it has not been recorded yet. |
int value__ |