MomentumOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements the Momentum optimization algorithm for gradient-based optimization.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → MomentumOptimizer<T, TInput, TOutput>
For Beginners
Imagine you’re rolling a ball down a hill to find the lowest point. The Momentum optimizer is like giving that ball some “memory” of its previous movements. This helps it move faster in consistent directions and resist getting stuck in small bumps or divots along the way.
How It Works
The Momentum optimizer is an extension of gradient descent that helps accelerate the optimization process in relevant directions and dampens oscillations. It does this by adding a fraction of the update vector of the past time step to the current update vector.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new MomentumOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with MomentumOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
MomentumOptimizer(IFullModel<T, TInput, TOutput> model, MomentumOptimizerOptions<T, TInput, TOutput> options) |
Initializes a new instance of the MomentumOptimizer class. |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuUpdate { get; } |
Gets whether this optimizer supports GPU-accelerated parameter updates. |
Methods
| Method | Summary |
|---|---|
MomentumOptimizer<T, TInput, TOutput> CreateForFunction(MomentumOptimizerOptions<T, TInput, TOutput> options) |
Creates a momentum optimizer for minimizing a plain function, with no model attached. |
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Gets the current optimization algorithm options. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the Momentum algorithm. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses a momentum-based gradient update to recover original parameters. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the Momentum optimization algorithm. |
void DisposeGpuState() |
Disposes GPU-allocated optimizer state. |
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) |
Initializes Momentum optimizer state on the GPU. |
void Reset() |
Resets the legacy vector, per-parameter tape, and device-resident velocity state. |
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) |
Updates parameters on the GPU using the SGD with momentum kernel. |