ScaledAdamOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
ScaledAdam (k2-fsa/icefall, as GPT-SoVITS vendors it in AR/modules/optim.py): Adam whose update to each
tensor is scaled by that tensor’s root-mean-square, which also learns each tensor’s overall scale, with momentum kept
as a parameter change and gradients clipped against the recent median gradient norm.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → ScaledAdamOptimizer<T, TInput, TOutput>
How It Works
Per tensor with more than one element: δ ← β₁δ; its scale gradient Σ θ·g is recorded each step and every
size_update_period steps the recorded rms is refreshed and the scale takes an Adam-like step
(δ += (1 − β₁)·θ·s, held at zero below param_min_rms and shrinking above param_max_rms); then
v ← β₂v + (1 − β₂)g², bias-corrected while the correction is below 0.99, and
δ += −lr(1 − β₁)·max(rms, param_min_rms)·g/(√v + ε); θ += δ. A scalar takes plain Adam without a first
moment at lr·scalar_lr_scale, clamped to ±scalar_max before its change is added.
Clipping, when enabled, records the norm of the rms-scaled gradients each step and every
clipping_update_period steps sets the threshold to clipping_scale times the median; from then on the
returned factor is min(1, threshold/norm). As in the reference, the factor scales only the gradient the scale
learning records; the Adam and scalar updates read the unscaled gradient.
The reference stacks tensors of the same shape into batches; every quantity is per tensor (or global), so the tensors are kept separate here with the same result.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new ScaledAdamOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with ScaledAdamOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
ScaledAdamOptimizer(IFullModel<T, TInput, TOutput> model, ScaledAdamOptimizerOptions<T, TInput, TOutput> options) |
Creates the optimizer. |
Properties
| Property | Summary |
|---|---|
int StepCount { get; } |
The steps taken. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
|
void Reset() |