NoamHoldAnnealingScheduler
Models & Types ยท AiDotNet.LearningRateSchedulers
Implements the extended Noam schedule introduced for Squeezeformer.
Inheritance: LearningRateSchedulerBase โ NoamHoldAnnealingScheduler
How It Works
For one-based optimizer step t, peak learning rate p, warmup
length T0, hold length Th, and decay power d:
The ordinary Noam decay is the special case Th = 0, d = 0.5.
Squeezeformer uses 20 epochs of warmup, 160 epochs of hold, and d = 1.
Constructors
| Constructor | Summary |
|---|---|
NoamHoldAnnealingScheduler(double peakLearningRate, int warmupSteps, int holdSteps, double decayRate) |
Initializes an extended Noam warmup-hold-decay schedule. |
Properties
| Property | Summary |
|---|---|
double DecayRate { get; } |
Gets the polynomial decay power. |
double PeakLearningRate { get; } |
Gets the peak learning rate. |
int HoldSteps { get; } |
Gets the number of optimizer steps that hold the peak rate. |
int WarmupSteps { get; } |
Gets the number of linear-warmup optimizer steps. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
void Reset() |