CosineAnnealingWarmRestartsScheduler

Models & Types · AiDotNet.LearningRateSchedulers

Sets the learning rate using cosine annealing with warm restarts.

Inheritance: LearningRateSchedulerBase → CosineAnnealingWarmRestartsScheduler

For Beginners

Imagine running a race in sprints instead of one continuous run. After each sprint (cycle), you rest (restart learning rate) and then sprint again. This “warm restart” approach helps the model escape local minima and often finds better solutions. The sprints can optionally get longer each time (controlled by T_mult), allowing for more fine-tuning in later cycles.

How It Works

This scheduler implements the SGDR (Stochastic Gradient Descent with Warm Restarts) algorithm. It uses cosine annealing but periodically restarts the learning rate to the initial value, optionally increasing the period between restarts.

Based on the paper “SGDR: Stochastic Gradient Descent with Warm Restarts” by Loshchilov & Hutter.

Example

using AiDotNet.LearningRateSchedulers;
// Warm restarts with initial period of 10, doubling each cycle
var scheduler = new CosineAnnealingWarmRestartsScheduler(
    baseLearningRate: 0.1,
    t0: 10,
    tMult: 2,
    etaMin: 0.001
);

Constructors

Constructor Summary
CosineAnnealingWarmRestartsScheduler(double baseLearningRate, int t0, int tMult, double etaMin) Initializes a new instance of the CosineAnnealingWarmRestartsScheduler class.

Properties

Property Summary
double EtaMin { get; } Gets the minimum learning rate.
int CurrentCycle { get; } Gets the current cycle number.
int T0 { get; } Gets the initial period.
int TMult { get; } Gets the period multiplier.

Methods

Method Summary
Dictionary<string, object> GetState()
double Step()
void LoadState(Dictionary<string, object> state)
void Reset()