Learning Rate Schedulers

29 public types in this namespace, organized by kind.

Models & Types (18)

Type Summary
AdaptiveFitnessScheduler Adjusts the learning rate in response to whether the monitored metric improved: shrink while improving, grow while stalled.
ConstantLRScheduler Maintains a constant learning rate throughout training.
CosineAnnealingLRScheduler Sets the learning rate using a cosine annealing schedule.
CosineAnnealingWarmRestartsScheduler Sets the learning rate using cosine annealing with warm restarts.
CyclicLRScheduler Implements cyclical learning rate policy.
ExponentialLRScheduler Decays the learning rate exponentially every step.
LambdaLRScheduler Sets the learning rate using a user-defined lambda function.
LinearWarmupScheduler Implements linear learning rate warmup followed by constant or decay schedule.
MultiStepLRScheduler Decays the learning rate by gamma at each milestone step.
NoamHoldAnnealingScheduler Implements the extended Noam schedule introduced for Squeezeformer.
NoamSchedule Implements the Noam learning rate schedule from “Attention Is All You Need” (Vaswani et al.
OneCycleLRScheduler Implements the 1cycle learning rate policy.
PolynomialLRScheduler Decays the learning rate using a polynomial function.
ReduceOnPlateauScheduler Reduces learning rate when a metric has stopped improving.
SequentialLRScheduler Chains multiple learning rate schedulers together in sequence.
StepLRScheduler Decays the learning rate by a factor (gamma) every specified number of steps.
TriStageScheduler Implements the tri-stage schedule: linear warmup, then a constant hold, then linear decay.
WarmupStableDecayScheduler Warmup-Stable-Decay (WSD) learning-rate schedule: a linear warmup, a long constant (“stable”) phase, and a final decay to endLr.

Base Classes (1)

Type Summary
LearningRateSchedulerBase Base class for learning rate schedulers providing common functionality.

Interfaces (1)

Type Summary
ILearningRateScheduler Interface for learning rate schedulers that adjust the learning rate during training.

Enums (8)

Type Summary
AnnealingStrategy Annealing strategy for the decay phase.
CyclicMode Mode for cyclic learning rate.
DecayMode Decay mode after warmup phase.
DecayShape Shape of the final decay phase.
LearningRateSchedulerType Enumeration of available learning rate scheduler types.
Mode Optimization mode.
SchedulerStepMode Specifies when the learning rate scheduler should be stepped during training.
ThresholdMode Threshold comparison mode.

Helpers & Utilities (1)

Type Summary
LearningRateSchedulerFactory Factory for creating learning rate schedulers with common configurations.