AdamOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

Implements the Adam (Adaptive Moment Estimation) optimization algorithm for gradient-based optimization.

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AdamOptimizer<T, TInput, TOutput>

For Beginners

Adam is like a smart personal trainer for your machine learning model. It helps your model learn efficiently by adjusting how it learns based on past experiences.

How It Works

Adam is an advanced optimization algorithm that combines ideas from RMSprop and Momentum optimization methods. It adapts the learning rates for each parameter individually and is well-suited for problems with noisy or sparse gradients.

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdamOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with AdamOptimizer.");

Constructors

Constructor Summary
AdamOptimizer(IFullModel<T, TInput, TOutput> model, AdamOptimizerOptions<T, TInput, TOutput> options) Initializes a new instance of the AdamOptimizer class.

Properties

Property Summary
bool SupportsGpuUpdate { get; } Gets whether this optimizer supports GPU-accelerated parameter updates.

Methods

Method Summary
AdamOptimizer<T, TInput, TOutput> CreateForFunction(AdamOptimizerOptions<T, TInput, TOutput> options) Creates an Adam optimizer for minimizing a plain function, with no model attached.
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) Updates a matrix of parameters using the Adam optimization algorithm.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() Gets the current optimizer options.
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) Performs the optimization process using the Adam algorithm.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses an Adam gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a matrix of parameters using the Adam optimization algorithm.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) Initializes Adam optimizer state on the GPU.
void Reset() Resets the optimizer’s internal state.
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) Updates parameters on the GPU using the Adam kernel.