LinearWarmupScheduler
Models & Types · AiDotNet.LearningRateSchedulers
Implements linear learning rate warmup followed by constant or decay schedule.
Inheritance: LearningRateSchedulerBase → LinearWarmupScheduler
For Beginners
When training starts, the model’s weights are random and can produce large, unstable gradients. Starting with a very small learning rate and gradually increasing it (warmup) helps the model stabilize before moving to the full learning rate. Think of it like warming up an engine before driving at full speed.
How It Works
Linear warmup gradually increases the learning rate from a small initial value to the target learning rate over a specified number of warmup steps. This is commonly used in transformer training and helps stabilize early training dynamics.
This scheduler supports three modes after warmup:
- Constant: Keep the base learning rate after warmup
- Linear decay: Linearly decrease to a minimum value
- Cosine decay: Use cosine annealing to decrease to a minimum value
Example
using AiDotNet.LearningRateSchedulers;
// Warmup for 1000 steps, then linear decay over remaining 9000 steps
var scheduler = new LinearWarmupScheduler(
baseLearningRate: 0.001,
warmupSteps: 1000,
totalSteps: 10000,
decayMode: LinearWarmupScheduler.DecayMode.Linear
);
Constructors
| Constructor | Summary |
|---|---|
LinearWarmupScheduler(double baseLearningRate, int warmupSteps, int totalSteps, double warmupInitLr, DecayMode? decayMode, double endLr) |
Initializes a new instance of the LinearWarmupScheduler class. |
Properties
| Property | Summary |
|---|---|
DecayMode CurrentDecayMode { get; } |
Gets the decay mode. |
double EndLr { get; } |
Gets the final learning rate reached after the decay phase. |
double WarmupInitLr { get; } |
Gets the learning rate at the start of warmup (warmup-init LR). |
int TotalSteps { get; } |
Gets the total number of steps. |
int WarmupSteps { get; } |
Gets the number of warmup steps. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
void Reset() |