CyclicLRScheduler

Models & Types · AiDotNet.LearningRateSchedulers

Implements cyclical learning rate policy.

Inheritance: LearningRateSchedulerBase → CyclicLRScheduler

For Beginners

Instead of always decreasing the learning rate, cyclic learning rates go up and down in cycles. The idea is that periodically increasing the learning rate can help the model escape local minima (suboptimal solutions) and explore better solutions. Think of it like occasionally taking bigger jumps while hiking to avoid getting stuck in small valleys.

How It Works

CyclicLR cycles the learning rate between two boundaries with a constant frequency. This approach can help escape local minima and find better solutions by periodically increasing the learning rate.

Based on the paper “Cyclical Learning Rates for Training Neural Networks” by Leslie N. Smith.

Example

using AiDotNet.LearningRateSchedulers;
// Triangular mode cycling between 0.001 and 0.1
var scheduler = new CyclicLRScheduler(
    baseLearningRate: 0.001,
    maxLearningRate: 0.1,
    stepSizeUp: 2000,
    mode: CyclicLRScheduler.CyclicMode.Triangular
);

Constructors

Constructor Summary
CyclicLRScheduler(double baseLearningRate, double maxLearningRate, int stepSizeUp, int? stepSizeDown, CyclicMode mode, double gamma) Initializes a new instance of the CyclicLRScheduler class.

Properties

Property Summary
CyclicMode Mode { get; } Cyclic mode (Triangular / Triangular2 / ExponentialRange).
double MaxLearningRate { get; } Gets the maximum learning rate.
int CycleCount { get; } Gets the current cycle count.
int StepSizeDown { get; } Gets the step size for decreasing phase.
int StepSizeUp { get; } Gets the step size for increasing phase.

Methods

Method Summary
Dictionary<string, object> GetState()