NesterovAcceleratedGradientOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements the Nesterov Accelerated Gradient optimization algorithm.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → NesterovAcceleratedGradientOptimizer<T, TInput, TOutput>
For Beginners
Imagine you’re skiing down a hill. Regular gradient descent is like looking at your current position to decide where to go next. NAG is like looking ahead to where you’ll be after your next move, and then deciding how to adjust your path. This “look-ahead” helps you navigate the slope more efficiently, especially around tricky turns.
How It Works
The Nesterov Accelerated Gradient (NAG) is an optimization algorithm that improves upon standard gradient descent. It introduces a smart prediction of the next position of the parameters, which helps to dampen oscillations and improve convergence, especially in scenarios with high curvature or small but consistent gradients.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new NesterovAcceleratedGradientOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with NesterovAcceleratedGradientOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
NesterovAcceleratedGradientOptimizer(IFullModel<T, TInput, TOutput> model, NesterovAcceleratedGradientOptimizerOptions<T, TInput, TOutput> options, IEngine engine) |
Initializes a new instance of the NesterovAcceleratedGradientOptimizer class. |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuUpdate { get; } |
Gets whether this optimizer supports GPU-accelerated parameter updates. |
Methods
| Method | Summary |
|---|---|
NesterovAcceleratedGradientOptimizer<T, TInput, TOutput> CreateForFunction(NesterovAcceleratedGradientOptimizerOptions<T, TInput, TOutput> options) |
Creates a Nesterov accelerated gradient optimizer for minimizing a plain function, with no model attached. |
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Gets the current options of the Nesterov Accelerated Gradient optimizer. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the Nesterov Accelerated Gradient algorithm. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses a Nesterov Accelerated Gradient update to recover original parameters. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the Nesterov Accelerated Gradient algorithm. |
void DisposeGpuState() |
Disposes GPU-allocated optimizer state. |
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) |
Initializes NAG optimizer state on the GPU. |
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) |
Updates parameters on the GPU using the NAG kernel. |