AdamWOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
AdamW with compressed moments (MomentStorage): bfloat16 or
8-bit block-wise moments, the same state on the device and the host.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AdamWOptimizer<T, TInput, TOutput>
How It Works
Each parameter’s moments live in one CompressedMomentSlot, either as device buffers (the GPU-resident
step, TryAdamBf16Step / GpuOptimizer.TryAdam8BitStep) or as host arrays (the host
step, HostCompressedMomentAdam). Both use one byte layout, so a slot moves between them by upload or
download, and the two steps produce the same bits.
Checkpoints reuse the tape-state serializer, packing and releasing one tensor at a time as exact float values (bf16 widens exactly; codebook indices 0-255 are exact; block scales occupy the leading entries). A versioned header validates format-affecting options before restored state is committed.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new AdamWOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with AdamWOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
AdamWOptimizer(IFullModel<T, TInput, TOutput> model, AdamWOptimizerOptions<T, TInput, TOutput> options) |
Initializes a new instance of the AdamWOptimizer class. |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuUpdate { get; } |
Gets whether this optimizer supports GPU-accelerated parameter updates. |
bool UseAMSGrad { get; } |
Gets whether AMSGrad variant is enabled. |
double WeightDecay { get; } |
Gets the current weight decay coefficient. |
Methods
| Method | Summary |
|---|---|
AdamWOptimizer<T, TInput, TOutput> CreateForFunction(AdamWOptimizerOptions<T, TInput, TOutput> options) |
Creates an AdamW optimizer for minimizing a plain function, with no model attached. |
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) |
Updates a matrix of parameters using the AdamW optimization algorithm. |
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Gets the current optimizer options. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the optimization process using the AdamW algorithm. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses an AdamW gradient update to recover original parameters. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a matrix of parameters using the AdamW optimization algorithm. |
void DisposeGpuState() |
Disposes GPU-allocated optimizer state. |
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) |
Initializes AdamW optimizer state on the GPU. |
void Reset() |
Resets the optimizer’s internal state. |
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) |
Updates parameters on the GPU using the AdamW kernel. |