AMSGradBiasCorrection
Enums · AiDotNet.Enums
Which bias corrections AMSGrad applies to its moment estimates.
For Beginners
Both moving averages start at zero, so early on they are biased toward zero.
“Bias correction” scales them up to compensate. The original paper skips that step; PyTorch includes it, which
makes the first few updates larger. Pick Paper to reproduce the paper, or PyTorch to
match results from PyTorch.
How It Works
AMSGrad (Reddi, Kale and Kumar, “On the Convergence of Adam and Beyond”, ICLR 2018) keeps a running maximum of
the second moment so the effective learning rate never increases. The paper’s Algorithm 2 applies no bias
correction at all. Most frameworks instead layer AMSGrad onto Adam’s bias correction: PyTorch’s
Adam(amsgrad=True) divides the first moment by (1 - beta1^t) and the running maximum by (1 - beta2^t).
Fields
| Field | Summary |
|---|---|
AMSGradBiasCorrection Paper |
No bias correction, as in the paper’s Algorithm 2: update = lr * m / (sqrt(vMax) + epsilon). |
AMSGradBiasCorrection PyTorch |
PyTorch amsgrad=True: update = lr * (m / (1 - beta1^t)) / (sqrt(vMax / (1 - beta2^t)) + epsilon). |
int value__ |