AMSGradBiasCorrection

Enums · AiDotNet.Enums

Which bias corrections AMSGrad applies to its moment estimates.

For Beginners

Both moving averages start at zero, so early on they are biased toward zero. “Bias correction” scales them up to compensate. The original paper skips that step; PyTorch includes it, which makes the first few updates larger. Pick Paper to reproduce the paper, or PyTorch to match results from PyTorch.

How It Works

AMSGrad (Reddi, Kale and Kumar, “On the Convergence of Adam and Beyond”, ICLR 2018) keeps a running maximum of the second moment so the effective learning rate never increases. The paper’s Algorithm 2 applies no bias correction at all. Most frameworks instead layer AMSGrad onto Adam’s bias correction: PyTorch’s Adam(amsgrad=True) divides the first moment by (1 - beta1^t) and the running maximum by (1 - beta2^t).

Fields

Field Summary
AMSGradBiasCorrection Paper No bias correction, as in the paper’s Algorithm 2: update = lr * m / (sqrt(vMax) + epsilon).
AMSGradBiasCorrection PyTorch PyTorch amsgrad=True: update = lr * (m / (1 - beta1^t)) / (sqrt(vMax / (1 - beta2^t)) + epsilon).
int value__