AMSGradOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements the AMSGrad optimization algorithm, an improved version of Adam optimizer.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AMSGradOptimizer<T, TInput, TOutput>
For Beginners
AMSGrad is like a smart assistant that helps adjust the learning process. It remembers past information to make better decisions about how quickly to learn in different parts of the problem.
How It Works
AMSGrad is an adaptive learning rate optimization algorithm that addresses some of the convergence issues in Adam. It maintains the maximum of past squared gradients to ensure non-decreasing step sizes.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new AMSGradOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with AMSGradOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
AMSGradOptimizer(IFullModel<T, TInput, TOutput> model, AMSGradOptimizerOptions<T, TInput, TOutput> options, IEngine engine) |
Initializes a new instance of the AMSGradOptimizer class. |
Methods
| Method | Summary |
|---|---|
AMSGradOptimizer<T, TInput, TOutput> CreateForFunction(AMSGradOptimizerOptions<T, TInput, TOutput> options) |
Creates an AMSGrad optimizer for minimizing a plain function, with no model attached. |
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Retrieves the current options of the optimizer. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the AMSGrad algorithm. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses an AMSGrad gradient update to recover original parameters. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the AMSGrad optimization algorithm. |
void DisposeGpuState() |
Disposes GPU-allocated optimizer state. |
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) |
|
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) |
GPU-accelerated parameter update for AMSGrad optimizer. |