DAdaptAdamOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
D-Adaptation Adam (Defazio and Mishchenko 2023, “Learning-Rate-Free Learning by D-Adaptation”): Adam whose step size d·lr comes from a running lower bound on the distance to the solution, estimated from the iterates.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → DAdaptAdamOptimizer<T, TInput, TOutput>
For Beginners
Adam with a step size the optimizer works out for itself.
How It Works
Each step, over all parameters together: m ← β₁m + d·lr(1 − β₁)g, v ← β₂v + (1 − β₂)g²,
s ← β₂s + d·lr(1 − β₂)g, and the weighted sums Σ g²/(√v + ε), Σ s²/(√v + ε) and Σ|s|; then
gsq ← β₂·gsq + (d·lr)²(1 − β₂)Σ g²/(√v + ε), d̂ = (Σ s²/(√v + ε)/(1 − β₂) − gsq)/Σ|s|,
d ← max(d, min(d̂, d·growth)); then each parameter takes θ ← θ − wd·d·lr·θ − m/(√v + ε). The update
follows the reference implementation exactly (facebookresearch/dadaptation DAdaptAdam, vendored in
audiocraft as audiocraft/optim/dadam.py), including returning without an update while Σ|s| is zero.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new DAdaptAdamOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with DAdaptAdamOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
DAdaptAdamOptimizer(IFullModel<T, TInput, TOutput> model, DAdaptAdamOptimizerOptions<T, TInput, TOutput> options) |
Creates the optimizer. |
Properties
| Property | Summary |
|---|---|
double Distance { get; } |
The current distance estimate d. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
|
void Reset() |