OneCycleLRScheduler

Models & Types · AiDotNet.LearningRateSchedulers

Implements the 1cycle learning rate policy.

Inheritance: LearningRateSchedulerBase → OneCycleLRScheduler

For Beginners

The 1cycle policy is like warming up before a workout, going full intensity during the workout, and then cooling down. The learning rate starts low (warmup), ramps up to a maximum (peak training), and then decreases to very low values (fine-tuning). This approach often allows training with higher maximum learning rates and can achieve better results in fewer epochs.

How It Works

The 1cycle policy starts with a low learning rate, increases it to a maximum, then decreases it again. This approach has been shown to enable faster training and better final performance, especially when combined with momentum cycling.

Based on the paper “Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates” by Leslie N. Smith and Nicholay Topin.

Example

using AiDotNet.LearningRateSchedulers;
// 1cycle policy over 100 epochs with peak LR of 0.1
var scheduler = new OneCycleLRScheduler(
    maxLearningRate: 0.1,
    totalSteps: 100,
    pctStart: 0.3,      // 30% warmup
    divFactor: 25,      // Start LR = 0.1/25 = 0.004
    finalDivFactor: 1e4 // End LR = 0.1/10000 = 0.00001
);

Constructors

Constructor Summary
OneCycleLRScheduler(double maxLearningRate, int totalSteps, double pctStart, double divFactor, double finalDivFactor, AnnealingStrategy annealStrategy) Initializes a new instance of the OneCycleLRScheduler class.

Properties

Property Summary
double MaxLearningRate { get; } Gets the maximum learning rate.
double PctStart { get; } Gets the percentage of steps for warmup.
int TotalSteps { get; } Gets the total number of steps.

Methods

Method Summary
Dictionary<string, object> GetState()