AdamWOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

AdamW with compressed moments (MomentStorage): bfloat16 or 8-bit block-wise moments, the same state on the device and the host.

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AdamWOptimizer<T, TInput, TOutput>

How It Works

Each parameter’s moments live in one CompressedMomentSlot, either as device buffers (the GPU-resident step, TryAdamBf16Step / GpuOptimizer.TryAdam8BitStep) or as host arrays (the host step, HostCompressedMomentAdam). Both use one byte layout, so a slot moves between them by upload or download, and the two steps produce the same bits.

Checkpoints reuse the tape-state serializer, packing and releasing one tensor at a time as exact float values (bf16 widens exactly; codebook indices 0-255 are exact; block scales occupy the leading entries). A versioned header validates format-affecting options before restored state is committed.

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdamWOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with AdamWOptimizer.");

Constructors

Constructor Summary
AdamWOptimizer(IFullModel<T, TInput, TOutput> model, AdamWOptimizerOptions<T, TInput, TOutput> options) Initializes a new instance of the AdamWOptimizer class.

Properties

Property Summary
bool SupportsGpuUpdate { get; } Gets whether this optimizer supports GPU-accelerated parameter updates.
bool UseAMSGrad { get; } Gets whether AMSGrad variant is enabled.
double WeightDecay { get; } Gets the current weight decay coefficient.

Methods

Method Summary
AdamWOptimizer<T, TInput, TOutput> CreateForFunction(AdamWOptimizerOptions<T, TInput, TOutput> options) Creates an AdamW optimizer for minimizing a plain function, with no model attached.
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) Updates a matrix of parameters using the AdamW optimization algorithm.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() Gets the current optimizer options.
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) Performs the optimization process using the AdamW algorithm.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses an AdamW gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a matrix of parameters using the AdamW optimization algorithm.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) Initializes AdamW optimizer state on the GPU.
void Reset() Resets the optimizer’s internal state.
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) Updates parameters on the GPU using the AdamW kernel.