RAdamOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements RAdam (Rectified Adam), the variant of Adam that rectifies the variance of the adaptive learning rate instead of relying on a hand-tuned warmup schedule.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → RAdamOptimizer<T, TInput, TOutput>
For Beginners
Adam decides how big a step to take for each individual parameter by looking at how much that parameter’s gradient has been bouncing around. Early in training there are only a few gradients to judge from, so that measurement is unreliable – and acting on an unreliable measurement is what makes Adam unstable at the start. The usual fix is “warmup”, where you manually start with a tiny learning rate and ramp it up. RAdam works out mathematically how trustworthy the measurement currently is and scales the step by that, and for the first few steps – when it is not trustworthy at all – it just takes an ordinary momentum step instead. The result behaves like Adam-with-warmup without you having to tune a warmup schedule.
How It Works
Faithful to Algorithm 2 of “On the Variance of the Adaptive Learning Rate and Beyond” (Liu, Jiang, He, Chen, Liu, Gao and Han; ICLR 2020; arXiv:1908.03265). Per step t:
Both branches matter. The else branch is not a degenerate case to be optimized away: for the first
few steps the variance of the adaptive term is not merely large but undefined, so the algorithm deliberately
takes a plain (bias-corrected) momentum step with NO adaptive scaling at all. At the default beta2 = 0.999
that covers roughly the first four steps. Collapsing the two branches into one is the single easiest way to
get RAdam subtly wrong, because the resulting optimizer still trains – just without the property the paper
exists to provide.
Epsilon is a numerical guard on the rectified branch only, matching the reference implementation. The paper’s l_t has no epsilon, and the un-rectified branch has no denominator to guard.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new RAdamOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with RAdamOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
RAdamOptimizer(IFullModel<T, TInput, TOutput> model, RAdamOptimizerOptions<T, TInput, TOutput> options) |
Initializes a new instance of the RAdamOptimizer class. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Retrieves the current options of the optimizer. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the RAdam algorithm. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses a RAdam gradient update to recover the parameters from before the step. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the RAdam optimization algorithm. |