NoamSchedule
Models & Types · AiDotNet.LearningRateSchedulers
Implements the Noam learning rate schedule from “Attention Is All You Need” (Vaswani et al. 2017, §5.3): inverse-square-root decay with linear warmup.
Inheritance: LearningRateSchedulerBase → NoamSchedule
For Beginners
Transformers are sensitive to learning-rate choice early in training because attention weights are softmax-normalized and gradients can explode when the network has to figure out which tokens to attend to. The Noam schedule ramps up the learning rate slowly for the first few thousand steps (the “warmup”), then decreases it like the inverse square root of the step number. This is how every Transformer in the original 2017 paper was trained.
How It Works
Formula:
where t is the 1-indexed training step from the paper. Peaks at t = warmup
with value factor · d_model^(-0.5) · warmup^(-0.5), then decays as
t^(-0.5).
Step-counter convention (matches PyTorch / HuggingFace transformer
schedulers): the library’s _currentStep
is incremented at end-of-batch and represents “batches completed so far”
(0-based). The Noam paper’s t is 1-based, so this scheduler maps
t = step + 1 internally:
- Before any Step() call,
_currentStep = 0⇒t = 1⇒ warmup-start LR. - Batch N reads the LR that was set by the (N-1)th Step() call ⇒ lr(t=N) ⇒
t = step + 1withstep = N-1. - Reset restores the warmup-start LR (NOT
_baseLearningRate, which we use as a peak-LR sentinel for the base ctor’s positive guard).
This schedule pairs with Adam β₁=0.9, β₂=0.98, ε=1e-9 (the Vaswani 2017 hyperparameters): the small β₂ tracks rapidly-changing attention/embedding gradients, and the warmup phase keeps the initial updates small while the second-moment estimates have not yet stabilized. Without warmup, β₂=0.98 is too aggressive for early-training stability — which is why these values must be applied together, not piecewise.
Constructors
| Constructor | Summary |
|---|---|
NoamSchedule(int modelDimension, int warmupSteps, double factor) |
Initializes a new Noam schedule (Vaswani 2017 inverse-sqrt with linear warmup). |
Properties
| Property | Summary |
|---|---|
double Factor { get; } |
Multiplicative scale on the schedule (1.0 = paper-faithful). |
int ModelDimension { get; } |
Model dimension this schedule was configured for. |
int WarmupSteps { get; } |
Number of warmup steps configured. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
void Reset() |
Restores the scheduler to its initial state. |