GradientBasedOptimizerBase<T, TInput, TOutput>

Base Classes ยท AiDotNet.Optimizers

Represents a base class for gradient-based optimization algorithms.

Inheritance: OptimizerBase<T, TInput, TOutput> โ†’ GradientBasedOptimizerBase<T, TInput, TOutput>

Implements: IFunctionOptimizer<T>, IGradientBasedOptimizer<T, TInput, TOutput>

Known derived types: ADMMOptimizer<T, TInput, TOutput>, AMSGradOptimizer<T, TInput, TOutput>, ASGDOptimizer<T, TInput, TOutput>, AdaDeltaOptimizer<T, TInput, TOutput>, AdaMaxOptimizer<T, TInput, TOutput>, AdafactorOptimizer<T, TInput, TOutput>, AdagradOptimizer<T, TInput, TOutput>, Adam8BitOptimizer<T, TInput, TOutput>, AdamOptimizer<T, TInput, TOutput>, AdamWOptimizer<T, TInput, TOutput>, BFGSOptimizer<T, TInput, TOutput>, ConjugateGradientOptimizer<T, TInput, TOutput>, CoordinateDescentOptimizer<T, TInput, TOutput>, DAdaptAdamOptimizer<T, TInput, TOutput>, DFPOptimizer<T, TInput, TOutput>, FTRLOptimizer<T, TInput, TOutput>, GradientDescentOptimizer<T, TInput, TOutput>, LAMBOptimizer<T, TInput, TOutput>, LARSOptimizer<T, TInput, TOutput>, LBFGSOptimizer<T, TInput, TOutput>, LevenbergMarquardtOptimizer<T, TInput, TOutput>, LionOptimizer<T, TInput, TOutput>, MiniBatchGradientDescentOptimizer<T, TInput, TOutput>, MomentumOptimizer<T, TInput, TOutput>, NadamOptimizer<T, TInput, TOutput>, NesterovAcceleratedGradientOptimizer<T, TInput, TOutput>, NewtonMethodOptimizer<T, TInput, TOutput>, ProximalGradientDescentOptimizer<T, TInput, TOutput>, RAdamOptimizer<T, TInput, TOutput>, RootMeanSquarePropagationOptimizer<T, TInput, TOutput>, RpropOptimizer<T, TInput, TOutput>, ScaledAdamOptimizer<T, TInput, TOutput>, ScheduleFreeAdamWOptimizer<T, TInput, TOutput>, StochasticGradientDescentOptimizer<T, TInput, TOutput>, TrustRegionOptimizer<T, TInput, TOutput>

For Beginners

Think of gradient-based optimization like finding the bottom of a valley:

  • You start at a random point on a hilly landscape (your initial model parameters)
  • You look around to see which way is steepest downhill (calculate the gradient)
  • You take a step in that direction (update the parameters)
  • You repeat this process until you reach the bottom of the valley (optimize the model)

This approach helps the model learn by gradually adjusting its parameters to minimize errors.

How It Works

Gradient-based optimizers use the gradient of the loss function to update the model parameters in a direction that minimizes the loss. This base class provides common functionality for various gradient-based optimization techniques.

Properties

Property Summary
ILearningRateScheduler LearningRateScheduler { get; } Gets the current learning rate scheduler, if one is configured.
IRegularization<T, TInput, TOutput> ActiveRegularization { get; } The active regularization applied during gradient updates.
SchedulerStepMode SchedulerStepMode { get; } Gets the current scheduler step mode.
Vector<T> LastComputedGradients { get; }
bool IsMixedPrecisionEnabled { get; } Gets whether mixed-precision training is enabled for this optimizer.
bool SupportsGpuUpdate { get; } Gets whether this optimizer supports GPU-accelerated parameter updates.
int CurrentEpoch { get; } Gets the current training epoch.
int CurrentStep { get; } Gets the current training step (batch count).

Methods

Method Summary
IFullModel<T, TInput, TOutput> ApplyGradients(Vector<T> gradients, IFullModel<T, TInput, TOutput> model)
IFullModel<T, TInput, TOutput> ApplyGradients(Vector<T> originalParameters, Vector<T> gradients, IFullModel<T, TInput, TOutput> model)
IFullModel<T, TInput, TOutput> ApplyGradients(Vector<T> originalParameters, Vector<T> gradients, IFullModel<T, TInput, TOutput> model, bool gradientsAlreadyUnscaled) Applies gradients to parameters with explicit control over whether gradients are already unscaled.
Matrix<T> UpdateParameters(Matrix<T> parameters, Matrix<T> gradient) Updates a vector of parameters based on the calculated gradient.
T GetGradientNorm() Gets the L2 norm of the last computed gradients.
Tensor<T> UpdateParameters(Tensor<T> parameters, Tensor<T> gradient) Updates a vector of parameters based on the calculated gradient.
Vector<T> Minimize(Vector<T> initialParameters, Func<Vector<T>, ValueTuple<T, Vector<T>>> objectiveAndGradient, int maxIterations, T tolerance)
Vector<T> Minimize(Vector<T> initialParameters, Func<Vector<T>, ValueTuple<T, Vector<T>>> objectiveAndGradient, int maxIterations, T tolerance, Func<Vector<T>, Vector<T>> projection) Minimizes a function subject to a feasible set, by projecting each iterate back onto that set after every step.
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) Reverses a gradient update to recover original parameters.
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) Updates a vector of parameters based on the calculated gradient.
bool AreGradientsExploding(double threshold) Checks if the current gradients are exhibiting exploding gradient behavior.
bool AreGradientsVanishing(double threshold) Checks if the current gradients are exhibiting vanishing gradient behavior.
double GetCurrentLearningRate() Gets the current learning rate being used by this optimizer.
double StepScheduler() Steps the learning rate scheduler and updates the current learning rate.
void DisposeGpuState() Disposes GPU-allocated optimizer state.
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) Initializes optimizer state on the GPU for a given parameter count.
void OnBatchEnd() Called at the end of each training batch to update scheduler state if applicable.
void OnEpochEnd() Called at the end of each training epoch to update scheduler state if applicable.
void Reset() Resets the optimizer to its initial state.
void Step(TapeStepContext<T> context)
void UpdateParameters(List<ILayer<T>> layers) Updates the parameters of the model based on the calculated gradients.
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) Updates parameters on the GPU using optimizer-specific GPU kernels.