Learning Rate Schedulers
29 public types in this namespace, organized by kind.
Models & Types (18)
| Type | Summary |
|---|---|
AdaptiveFitnessScheduler |
Adjusts the learning rate in response to whether the monitored metric improved: shrink while improving, grow while stalled. |
ConstantLRScheduler |
Maintains a constant learning rate throughout training. |
CosineAnnealingLRScheduler |
Sets the learning rate using a cosine annealing schedule. |
CosineAnnealingWarmRestartsScheduler |
Sets the learning rate using cosine annealing with warm restarts. |
CyclicLRScheduler |
Implements cyclical learning rate policy. |
ExponentialLRScheduler |
Decays the learning rate exponentially every step. |
LambdaLRScheduler |
Sets the learning rate using a user-defined lambda function. |
LinearWarmupScheduler |
Implements linear learning rate warmup followed by constant or decay schedule. |
MultiStepLRScheduler |
Decays the learning rate by gamma at each milestone step. |
NoamHoldAnnealingScheduler |
Implements the extended Noam schedule introduced for Squeezeformer. |
NoamSchedule |
Implements the Noam learning rate schedule from “Attention Is All You Need” (Vaswani et al. |
OneCycleLRScheduler |
Implements the 1cycle learning rate policy. |
PolynomialLRScheduler |
Decays the learning rate using a polynomial function. |
ReduceOnPlateauScheduler |
Reduces learning rate when a metric has stopped improving. |
SequentialLRScheduler |
Chains multiple learning rate schedulers together in sequence. |
StepLRScheduler |
Decays the learning rate by a factor (gamma) every specified number of steps. |
TriStageScheduler |
Implements the tri-stage schedule: linear warmup, then a constant hold, then linear decay. |
WarmupStableDecayScheduler |
Warmup-Stable-Decay (WSD) learning-rate schedule: a linear warmup, a long constant (“stable”) phase, and a final decay to endLr. |
Base Classes (1)
| Type | Summary |
|---|---|
LearningRateSchedulerBase |
Base class for learning rate schedulers providing common functionality. |
Interfaces (1)
| Type | Summary |
|---|---|
ILearningRateScheduler |
Interface for learning rate schedulers that adjust the learning rate during training. |
Enums (8)
| Type | Summary |
|---|---|
AnnealingStrategy |
Annealing strategy for the decay phase. |
CyclicMode |
Mode for cyclic learning rate. |
DecayMode |
Decay mode after warmup phase. |
DecayShape |
Shape of the final decay phase. |
LearningRateSchedulerType |
Enumeration of available learning rate scheduler types. |
Mode |
Optimization mode. |
SchedulerStepMode |
Specifies when the learning rate scheduler should be stepped during training. |
ThresholdMode |
Threshold comparison mode. |
Helpers & Utilities (1)
| Type | Summary |
|---|---|
LearningRateSchedulerFactory |
Factory for creating learning rate schedulers with common configurations. |