NoamHoldAnnealingScheduler

Models & Types ยท AiDotNet.LearningRateSchedulers

Implements the extended Noam schedule introduced for Squeezeformer.

Inheritance: LearningRateSchedulerBase โ†’ NoamHoldAnnealingScheduler

How It Works

For one-based optimizer step t, peak learning rate p, warmup length T0, hold length Th, and decay power d:

The ordinary Noam decay is the special case Th = 0, d = 0.5. Squeezeformer uses 20 epochs of warmup, 160 epochs of hold, and d = 1.

Constructors

Constructor Summary
NoamHoldAnnealingScheduler(double peakLearningRate, int warmupSteps, int holdSteps, double decayRate) Initializes an extended Noam warmup-hold-decay schedule.

Properties

Property Summary
double DecayRate { get; } Gets the polynomial decay power.
double PeakLearningRate { get; } Gets the peak learning rate.
int HoldSteps { get; } Gets the number of optimizer steps that hold the peak rate.
int WarmupSteps { get; } Gets the number of linear-warmup optimizer steps.

Methods

Method Summary
Dictionary<string, object> GetState()
void Reset()