CosineAnnealingWarmRestartsScheduler
Models & Types · AiDotNet.LearningRateSchedulers
Sets the learning rate using cosine annealing with warm restarts.
Inheritance: LearningRateSchedulerBase → CosineAnnealingWarmRestartsScheduler
For Beginners
Imagine running a race in sprints instead of one continuous run. After each sprint (cycle), you rest (restart learning rate) and then sprint again. This “warm restart” approach helps the model escape local minima and often finds better solutions. The sprints can optionally get longer each time (controlled by T_mult), allowing for more fine-tuning in later cycles.
How It Works
This scheduler implements the SGDR (Stochastic Gradient Descent with Warm Restarts) algorithm. It uses cosine annealing but periodically restarts the learning rate to the initial value, optionally increasing the period between restarts.
Based on the paper “SGDR: Stochastic Gradient Descent with Warm Restarts” by Loshchilov & Hutter.
Example
using AiDotNet.LearningRateSchedulers;
// Warm restarts with initial period of 10, doubling each cycle
var scheduler = new CosineAnnealingWarmRestartsScheduler(
baseLearningRate: 0.1,
t0: 10,
tMult: 2,
etaMin: 0.001
);
Constructors
| Constructor | Summary |
|---|---|
CosineAnnealingWarmRestartsScheduler(double baseLearningRate, int t0, int tMult, double etaMin) |
Initializes a new instance of the CosineAnnealingWarmRestartsScheduler class. |
Properties
| Property | Summary |
|---|---|
double EtaMin { get; } |
Gets the minimum learning rate. |
int CurrentCycle { get; } |
Gets the current cycle number. |
int T0 { get; } |
Gets the initial period. |
int TMult { get; } |
Gets the period multiplier. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetState() |
|
double Step() |
|
void LoadState(Dictionary<string, object> state) |
|
void Reset() |