AdafactorOptimizer<T, TInput, TOutput>

Models & Types · AiDotNet.Optimizers

Adafactor (Shazeer and Stern 2018): Adam-style adaptivity with O(n+m) state instead of O(nm).

Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → AdafactorOptimizer<T, TInput, TOutput>

For Beginners

Optimizers that adapt per-weight step sizes have to remember something about every weight, which for a large model can cost as much memory as the model itself. Adafactor remembers a summary per row and per column instead and reconstructs the rest, trading a little precision for the ability to train models that otherwise would not fit.

How It Works

Adam keeps a second-moment estimate per weight. Adafactor keeps, for a matrix, only a per-row and a per-column sum of squared gradients, and reconstructs an approximation of the full matrix as their outer product divided by their total. For an n by m weight that is n+m numbers rather than nm, which is what makes it the optimizer of choice for very large embedding and projection layers. Vectors are not factored — there is nothing to gain — and the paper keeps a full moment for them, as this does.

Two further departures from Adam, both load-bearing rather than incidental. There is no first moment by default, which is most of the memory saving. And the step is clipped by its own RMS rather than the gradient being clipped by norm, which is what lets the paper’s schedules run without warmup.

This implementation is deliberately plain: unlike the older optimizers here it has no GPU or sparse-embedding fast path. It is correct and readable rather than tuned, and the factored reduction it needs is not one of the engine’s existing tensor operations.

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
    for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
    trainY[new[] { i, i % 2 }] = 1.0;
}

var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdafactorOptimizer<double, Tensor<double>, Tensor<double>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with AdafactorOptimizer.");

Constructors

Constructor Summary
AdafactorOptimizer(IFullModel<T, TInput, TOutput> model, AdafactorOptimizerOptions<T, TInput, TOutput> options) Creates an Adafactor optimizer for a model.

Methods

Method Summary
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient)