Optimizers

51 public types in this namespace, organized by kind.

Models & Types (48)

Type Summary
ADMMOptimizer<T, TInput, TOutput> Implements the Alternating Direction Method of Multipliers (ADMM) optimization algorithm.
AMSGradOptimizer<T, TInput, TOutput> Implements the AMSGrad optimization algorithm, an improved version of Adam optimizer.
ASGDOptimizer<T, TInput, TOutput> Implements ASGD (Averaged Stochastic Gradient Descent): decayed SGD that additionally maintains a running average of the iterates, which is the quantity the method is named for and the one worth reading out.
AdaDeltaOptimizer<T, TInput, TOutput> Implements the AdaDelta optimization algorithm for training neural networks and other machine learning models.
AdaMaxOptimizer<T, TInput, TOutput> Represents an AdaMax optimizer, an extension of Adam that uses the infinity norm.
AdafactorOptimizer<T, TInput, TOutput> Adafactor (Shazeer and Stern 2018): Adam-style adaptivity with O(n+m) state instead of O(nm).
AdagradOptimizer<T, TInput, TOutput> Represents an Adagrad (Adaptive Gradient) optimizer for gradient-based optimization.
Adam8BitOptimizer<T, TInput, TOutput> Implements an 8-bit quantized Adam optimizer that reduces memory usage by storing optimizer states in 8-bit format.
AdamOptimizer<T, TInput, TOutput> Implements the Adam (Adaptive Moment Estimation) optimization algorithm for gradient-based optimization.
AdamWOptimizer<T, TInput, TOutput> AdamW with compressed moments (MomentStorage): bfloat16 or 8-bit block-wise moments, the same state on the device and the host.
AntColonyOptimizer<T, TInput, TOutput> Implements the Ant Colony Optimization algorithm for solving optimization problems.
BFGSOptimizer<T, TInput, TOutput> Implements the Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization algorithm.
BayesianOptimizer<T, TInput, TOutput> Represents a Bayesian Optimizer for optimization problems.
CMAESGeneration<T> One CMA-ES generation sampled by Ask: the candidate points to evaluate, plus what Tell needs to update the distribution from their fitness.
CMAESOptimizer<T, TInput, TOutput> Implements the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) optimization algorithm.
CMAESSearchState The complete state of a CMA-ES search between generations: the distribution’s mean, step size and covariance, the two evolution paths, and the generation count.
ConjugateGradientOptimizer<T, TInput, TOutput> Implements the Conjugate Gradient optimization algorithm for numerical optimization problems.
CoordinateDescentOptimizer<T, TInput, TOutput> Implements the Coordinate Descent optimization algorithm for numerical optimization problems.
DAdaptAdamOptimizer<T, TInput, TOutput> D-Adaptation Adam (Defazio and Mishchenko 2023, “Learning-Rate-Free Learning by D-Adaptation”): Adam whose step size d·lr comes from a running lower bound on the distance to the solution, estimated from the iterates.
DFPOptimizer<T, TInput, TOutput> Implements the Davidon-Fletcher-Powell (DFP) optimization algorithm for numerical optimization problems.
DifferentialEvolutionOptimizer<T, TInput, TOutput> Implements the Differential Evolution optimization algorithm for numerical optimization problems.
FTRLOptimizer<T, TInput, TOutput> Represents a Follow The Regularized Leader (FTRL) optimizer for machine learning models.
GeneticAlgorithmOptimizer<T, TInput, TOutput> Represents a Genetic Algorithm optimizer for machine learning models.
GradientDescentOptimizer<T, TInput, TOutput> Represents a Gradient Descent optimizer for machine learning models.
LAMBOptimizer<T, TInput, TOutput> Implements the LAMB (Layer-wise Adaptive Moments for Batch training) optimization algorithm.
LARSOptimizer<T, TInput, TOutput> Implements the LARS (Layer-wise Adaptive Rate Scaling) optimization algorithm.
LBFGSOptimizer<T, TInput, TOutput> Implements the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) optimization algorithm.
LevenbergMarquardtOptimizer<T, TInput, TOutput> Implements the Levenberg-Marquardt optimization algorithm for non-linear least squares problems.
LionOptimizer<T, TInput, TOutput> Implements the Lion (Evolved Sign Momentum) optimization algorithm for gradient-based optimization.
MiniBatchGradientDescentOptimizer<T, TInput, TOutput> Implements the Mini-Batch Gradient Descent optimization algorithm.
ModifiedGradientDescentOptimizer<T> Modified Gradient Descent optimizer for Hope architecture.
MomentumOptimizer<T, TInput, TOutput> Implements the Momentum optimization algorithm for gradient-based optimization.
NadamOptimizer<T, TInput, TOutput> Implements the Nesterov-accelerated Adaptive Moment Estimation (Nadam) optimization algorithm.
NelderMeadOptimizer<T, TInput, TOutput> Implements the Nelder-Mead optimization algorithm, also known as the downhill simplex method.
NesterovAcceleratedGradientOptimizer<T, TInput, TOutput> Implements the Nesterov Accelerated Gradient optimization algorithm.
NewtonMethodOptimizer<T, TInput, TOutput> Implements the Newton’s Method optimization algorithm.
NormalOptimizer<T, TInput, TOutput> Implements a normal optimization algorithm with adaptive parameters.
OptimizationDataBatcher<T, TInput, TOutput> Provides batch iteration utilities for optimization input data.
ParticleSwarmOptimizer<T, TInput, TOutput> Implements a Particle Swarm Optimization algorithm for finding optimal solutions.
ProximalGradientDescentOptimizer<T, TInput, TOutput> Implements a Proximal Gradient Descent optimization algorithm which combines gradient descent with regularization.
RAdamOptimizer<T, TInput, TOutput> Implements RAdam (Rectified Adam), the variant of Adam that rectifies the variance of the adaptive learning rate instead of relying on a hand-tuned warmup schedule.
RootMeanSquarePropagationOptimizer<T, TInput, TOutput> Implements the Root Mean Square Propagation (RMSProp) optimization algorithm, an adaptive learning rate method.
RpropOptimizer<T, TInput, TOutput> Implements Rprop (resilient backpropagation): a per-weight adaptive step size driven by the SIGN of the gradient, with the magnitude discarded entirely.
ScaledAdamOptimizer<T, TInput, TOutput> ScaledAdam (k2-fsa/icefall, as GPT-SoVITS vendors it in AR/modules/optim.py): Adam whose update to each tensor is scaled by that tensor’s root-mean-square, which also learns each tensor’s overall scale, with momentum kept as a parameter c…
ScheduleFreeAdamWOptimizer<T, TInput, TOutput> Schedule-Free AdamW (Defazio et al.
StochasticGradientDescentOptimizer<T, TInput, TOutput> Represents a Stochastic Gradient Descent (SGD) optimizer for machine learning models.
TabuSearchOptimizer<T, TInput, TOutput> Represents a Tabu Search optimizer for machine learning models.
TrustRegionOptimizer<T, TInput, TOutput> Implements the Trust Region optimization algorithm for machine learning models.

Base Classes (2)

Type Summary
GradientBasedOptimizerBase<T, TInput, TOutput> Represents a base class for gradient-based optimization algorithms.
OptimizerBase<T, TInput, TOutput> Represents the base class for all optimization algorithms, providing common functionality and interfaces.

Helpers & Utilities (1)

Type Summary
OptimizationDataBatcherExtensions Extension methods for optimization data batching.