WarmupStableDecayScheduler

Models & Types · AiDotNet.LearningRateSchedulers

Warmup-Stable-Decay (WSD) learning-rate schedule: a linear warmup, a long constant (“stable”) phase, and a final decay to endLr.

Inheritance: LearningRateSchedulerBase → WarmupStableDecayScheduler

For Beginners

The learning rate ramps up, holds steady for most of training, then drops to (almost) zero at the end. Because the “hold steady” part doesn’t know when training will end, you can decide later to train longer without restarting from scratch.

How It Works

Unlike cosine annealing, nothing in the warmup or stable phase depends on the total run length, so a run that is checkpointed during the stable phase can be extended by resuming it with a larger decayStartStep. Language-model pretraining uses this to read quality at several token budgets from one stable trajectory: each budget branches a short decay from a stable checkpoint.

Reference: Hu et al., “MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies”, 2024 (https://arxiv.org/abs/2404.06395).

Example

using AiDotNet.LearningRateSchedulers;
// 500 warmup steps, stable until step 8000, linear decay to 0 by step 10000.
var wsd = new WarmupStableDecayScheduler(3e-3, warmupSteps: 500, decayStartStep: 8000, decaySteps: 2000);

Constructors

Constructor Summary
WarmupStableDecayScheduler(double baseLearningRate, int warmupSteps, int decayStartStep, int decaySteps, DecayShape decayShape, double endLr) Creates a WSD schedule.

Properties

Property Summary
DecayShape Shape { get; } Gets the decay shape.
double EndLr { get; } Gets the final learning rate.
int DecayStartStep { get; } Gets the step at which the decay begins.
int DecaySteps { get; } Gets the length of the decay phase.
int WarmupSteps { get; } Gets the number of warmup steps.

Methods

Method Summary
Dictionary<string, object> GetState()
void Reset()