AMSGradOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

Implements the AMSGrad optimization algorithm, an improved version of Adam optimizer.

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AMSGradOptimizer<T, TInput, TOutput>

For Beginners

AMSGrad is like a smart assistant that helps adjust the learning process. It remembers past information to make better decisions about how quickly to learn in different parts of the problem.

How It Works

AMSGrad is an adaptive learning rate optimization algorithm that addresses some of the convergence issues in Adam. It maintains the maximum of past squared gradients to ensure non-decreasing step sizes.

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AMSGradOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with AMSGradOptimizer.");

Constructors

Constructor Summary
AMSGradOptimizer(IFullModel<T, TInput, TOutput> model, AMSGradOptimizerOptions<T, TInput, TOutput> options, IEngine engine) Initializes a new instance of the AMSGradOptimizer class.

Methods

Method Summary
AMSGradOptimizer<T, TInput, TOutput> CreateForFunction(AMSGradOptimizerOptions<T, TInput, TOutput> options) Creates an AMSGrad optimizer for minimizing a plain function, with no model attached.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() Retrieves the current options of the optimizer.
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) Performs the optimization process using the AMSGrad algorithm.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses an AMSGrad gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a vector of parameters using the AMSGrad optimization algorithm.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend)
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) GPU-accelerated parameter update for AMSGrad optimizer.