TriStageScheduler
Models & Types · AiDotNet.LearningRateSchedulers
Implements the tri-stage schedule: linear warmup, then a constant hold, then linear decay.
Inheritance: LearningRateSchedulerBase → TriStageScheduler
For Beginners
Three phases. The rate climbs from near zero so early, noisy gradients cannot wreck the weights; it then sits at its peak for the bulk of training, where most of the learning happens; and it finally falls back towards zero so the model settles instead of bouncing around a minimum.
How It Works
For total steps N, warmup fraction w, hold fraction h, peak rate p
and floor m, with W = wN and H = hN:
This is the schedule wav2vec 2.0 fine-tunes with — “a tri-state rate schedule where the learning rate is warmed up for the first 10% of updates, held constant for the next 40% and then linearly decayed for the remainder” (Baevski et al. 2020, Sec. 4.3) — and it is shared by much of the speech literature that followed it.
It is deliberately its own scheduler rather than an approximation by an existing one. The nearest available shapes are warmup-then-decay, which has no hold phase at all, and the Noam-hold schedule, whose third stage decays as a power of the step rather than linearly. Both produce a visibly different curve, and substituting a different curve for a published one is harder to notice than having no schedule at all.
Constructors
| Constructor | Summary |
|---|---|
TriStageScheduler(double baseLearningRate, int warmupSteps, int holdSteps, int totalSteps, double minLearningRate, double decayPower) |
Creates a tri-stage schedule from explicit step counts. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
TriStageScheduler FromFractions(double baseLearningRate, int totalSteps, double warmupFraction, double holdFraction, double minLearningRate, double decayPower) |
Creates a tri-stage schedule from the fractions a paper usually states. |
void Reset() |