Optimizers

Reference for the optimization algorithms and learning-rate schedulers in AiDotNet.

Optimizers are generic over <T, TInput, TOutput> and take the model they optimize: new AdamWOptimizer<float, Tensor<float>, Tensor<float>>(model). Tuning (learning rate, betas, weight decay) lives in the matching *Options. Plug an optimizer in with ConfigureOptimizer(...).

First-Order Optimizers

Second-Order Optimizers

Evolutionary Optimizers

Learning-Rate Schedulers

Schedulers implement ILearningRateScheduler and attach via ConfigureLearningRateScheduler(...).

Using an Optimizer

Configure the model first, then hand the same model to the optimizer.

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var rng = new Random(0);
var trainX = new Tensor<float>(new[] { 64, 16 });
var trainY = new Tensor<float>(new[] { 64, 3 });
for (int i = 0; i < 64; i++)
{
    for (int j = 0; j < 16; j++) trainX[new[] { i, j }] = (float)rng.NextDouble();
    trainY[new[] { i, i % 3 }] = 1f;
}

var model = new NeuralNetwork<float>(new NeuralNetworkArchitecture<float>(
    inputFeatures: 16, numClasses: 3, complexity: NetworkComplexity.Simple));

// Swap AdamWOptimizer for any optimizer above — they share the (model, options) shape.
var result = await new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdamWOptimizer<float, Tensor<float>, Tensor<float>>(model))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine($"Trained; output [{string.Join(", ", result.Predict(trainX).Shape)}]");

Tuning via Options

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.Models.Options;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var trainX = new Tensor<float>(new[] { 32, 8 });
var trainY = new Tensor<float>(new[] { 32, 2 });
for (int i = 0; i < 32; i++) { trainX[new[] { i, 0 }] = i / 32f; trainY[new[] { i, i % 2 }] = 1f; }

var model = new NeuralNetwork<float>(new NeuralNetworkArchitecture<float>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

// The learning rate and every other knob live on the options object.
var options = new AdamOptimizerOptions<float, Tensor<float>, Tensor<float>>
{
    InitialLearningRate = 3e-4,   // the learning rate
    Epsilon = 1e-8,
    MaxGradientNorm = 1.0,        // gradient clipping
};

var result = await new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdamOptimizer<float, Tensor<float>, Tensor<float>>(model, options))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with custom optimizer options.");

Learning-Rate Schedules

Decay the learning rate over training with a scheduler — CosineAnnealingLRScheduler, StepLRScheduler, and others live in AiDotNet.LearningRateSchedulers and attach via ConfigureLearningRateScheduler(...).

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.LearningRateSchedulers;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;

var trainX = new Tensor<float>(new[] { 32, 8 });
var trainY = new Tensor<float>(new[] { 32, 2 });
for (int i = 0; i < 32; i++) { trainX[new[] { i, 0 }] = i / 32f; trainY[new[] { i, i % 2 }] = 1f; }

var model = new NeuralNetwork<float>(new NeuralNetworkArchitecture<float>(
    inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));

var result = await new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(model)
    .ConfigureOptimizer(new AdamWOptimizer<float, Tensor<float>, Tensor<float>>(model))
    // Cosine decay from 3e-4 over 100 epochs; StepLRScheduler(baseLearningRate, stepSize, gamma) for step decay.
    .ConfigureLearningRateScheduler(new CosineAnnealingLRScheduler(baseLearningRate: 3e-4, tMax: 100))
    .ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
    .BuildAsync();

Console.WriteLine("Trained with a cosine-annealing learning-rate schedule.");

Optimizer Selection Guide

Hyperparameter Guidelines