CosineAnnealingLRScheduler

Models & Types · AiDotNet.LearningRateSchedulers

Sets the learning rate using a cosine annealing schedule.

Inheritance: LearningRateSchedulerBase → CosineAnnealingLRScheduler

For Beginners

Instead of making sudden drops in learning rate, cosine annealing provides a smooth, curved decrease that follows the shape of a cosine wave. The learning rate starts high, decreases slowly at first, then more rapidly in the middle, and finally slows down again as it approaches the minimum. This smooth transition often leads to better model performance than abrupt changes.

How It Works

CosineAnnealingLR uses a cosine function to smoothly decrease the learning rate from the initial value to a minimum value over a specified number of steps. This is widely used in modern deep learning and often outperforms step-based decay schedules.

Formula: lr = min_lr + 0.5 * (base_lr - min_lr) * (1 + cos(π * step / T_max))

Example

using AiDotNet.LearningRateSchedulers;
// Cosine annealing over 100 epochs
var scheduler = new CosineAnnealingLRScheduler(
    baseLearningRate: 0.1,
    tMax: 100,
    etaMin: 0.001
);

Constructors

Constructor Summary
CosineAnnealingLRScheduler(double baseLearningRate, int tMax, double etaMin) Initializes a new instance of the CosineAnnealingLRScheduler class.

Properties

Property Summary
double EtaMin { get; } Gets the minimum learning rate.
int TMax { get; } Gets the maximum number of steps.

Methods

Method Summary
Dictionary<string, object> GetState()