WarmupStableDecayScheduler
Models & Types · AiDotNet.LearningRateSchedulers
Warmup-Stable-Decay (WSD) learning-rate schedule: a linear warmup, a long constant (“stable”) phase, and a
final decay to endLr.
Inheritance: LearningRateSchedulerBase → WarmupStableDecayScheduler
For Beginners
The learning rate ramps up, holds steady for most of training, then drops to (almost) zero at the end. Because the “hold steady” part doesn’t know when training will end, you can decide later to train longer without restarting from scratch.
How It Works
Unlike cosine annealing, nothing in the warmup or stable phase depends on the total run length, so a run that
is checkpointed during the stable phase can be extended by resuming it with a larger
decayStartStep. Language-model pretraining uses this to read quality at several token budgets from one
stable trajectory: each budget branches a short decay from a stable checkpoint.
Reference: Hu et al., “MiniCPM: Unveiling the Potential of Small Language Models with Scalable
Training Strategies”, 2024 (https://arxiv.org/abs/2404.06395).
Example
using AiDotNet.LearningRateSchedulers;
// 500 warmup steps, stable until step 8000, linear decay to 0 by step 10000.
var wsd = new WarmupStableDecayScheduler(3e-3, warmupSteps: 500, decayStartStep: 8000, decaySteps: 2000);
Constructors
| Constructor | Summary |
|---|---|
WarmupStableDecayScheduler(double baseLearningRate, int warmupSteps, int decayStartStep, int decaySteps, DecayShape decayShape, double endLr) |
Creates a WSD schedule. |
Properties
| Property | Summary |
|---|---|
DecayShape Shape { get; } |
Gets the decay shape. |
double EndLr { get; } |
Gets the final learning rate. |
int DecayStartStep { get; } |
Gets the step at which the decay begins. |
int DecaySteps { get; } |
Gets the length of the decay phase. |
int WarmupSteps { get; } |
Gets the number of warmup steps. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
void Reset() |