ADMMOptimizer<T, TInput, TOutput> |
Implements the Alternating Direction Method of Multipliers (ADMM) optimization algorithm. |
AMSGradOptimizer<T, TInput, TOutput> |
Implements the AMSGrad optimization algorithm, an improved version of Adam optimizer. |
ASGDOptimizer<T, TInput, TOutput> |
Implements ASGD (Averaged Stochastic Gradient Descent): decayed SGD that additionally maintains a running average of the iterates, which is the quantity the method is named for and the one worth reading out. |
AdaDeltaOptimizer<T, TInput, TOutput> |
Implements the AdaDelta optimization algorithm for training neural networks and other machine learning models. |
AdaMaxOptimizer<T, TInput, TOutput> |
Represents an AdaMax optimizer, an extension of Adam that uses the infinity norm. |
AdafactorOptimizer<T, TInput, TOutput> |
Adafactor (Shazeer and Stern 2018): Adam-style adaptivity with O(n+m) state instead of O(nm). |
AdagradOptimizer<T, TInput, TOutput> |
Represents an Adagrad (Adaptive Gradient) optimizer for gradient-based optimization. |
Adam8BitOptimizer<T, TInput, TOutput> |
Implements an 8-bit quantized Adam optimizer that reduces memory usage by storing optimizer states in 8-bit format. |
AdamOptimizer<T, TInput, TOutput> |
Implements the Adam (Adaptive Moment Estimation) optimization algorithm for gradient-based optimization. |
AdamWOptimizer<T, TInput, TOutput> |
AdamW with compressed moments (MomentStorage): bfloat16 or 8-bit block-wise moments, the same state on the device and the host. |
AntColonyOptimizer<T, TInput, TOutput> |
Implements the Ant Colony Optimization algorithm for solving optimization problems. |
BFGSOptimizer<T, TInput, TOutput> |
Implements the Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization algorithm. |
BayesianOptimizer<T, TInput, TOutput> |
Represents a Bayesian Optimizer for optimization problems. |
CMAESGeneration<T> |
One CMA-ES generation sampled by Ask: the candidate points to evaluate, plus what Tell needs to update the distribution from their fitness. |
CMAESOptimizer<T, TInput, TOutput> |
Implements the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) optimization algorithm. |
CMAESSearchState |
The complete state of a CMA-ES search between generations: the distribution’s mean, step size and covariance, the two evolution paths, and the generation count. |
ConjugateGradientOptimizer<T, TInput, TOutput> |
Implements the Conjugate Gradient optimization algorithm for numerical optimization problems. |
CoordinateDescentOptimizer<T, TInput, TOutput> |
Implements the Coordinate Descent optimization algorithm for numerical optimization problems. |
DAdaptAdamOptimizer<T, TInput, TOutput> |
D-Adaptation Adam (Defazio and Mishchenko 2023, “Learning-Rate-Free Learning by D-Adaptation”): Adam whose step size d·lr comes from a running lower bound on the distance to the solution, estimated from the iterates. |
DFPOptimizer<T, TInput, TOutput> |
Implements the Davidon-Fletcher-Powell (DFP) optimization algorithm for numerical optimization problems. |
DifferentialEvolutionOptimizer<T, TInput, TOutput> |
Implements the Differential Evolution optimization algorithm for numerical optimization problems. |
FTRLOptimizer<T, TInput, TOutput> |
Represents a Follow The Regularized Leader (FTRL) optimizer for machine learning models. |
GeneticAlgorithmOptimizer<T, TInput, TOutput> |
Represents a Genetic Algorithm optimizer for machine learning models. |
GradientDescentOptimizer<T, TInput, TOutput> |
Represents a Gradient Descent optimizer for machine learning models. |
LAMBOptimizer<T, TInput, TOutput> |
Implements the LAMB (Layer-wise Adaptive Moments for Batch training) optimization algorithm. |
LARSOptimizer<T, TInput, TOutput> |
Implements the LARS (Layer-wise Adaptive Rate Scaling) optimization algorithm. |
LBFGSOptimizer<T, TInput, TOutput> |
Implements the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) optimization algorithm. |
LevenbergMarquardtOptimizer<T, TInput, TOutput> |
Implements the Levenberg-Marquardt optimization algorithm for non-linear least squares problems. |
LionOptimizer<T, TInput, TOutput> |
Implements the Lion (Evolved Sign Momentum) optimization algorithm for gradient-based optimization. |
MiniBatchGradientDescentOptimizer<T, TInput, TOutput> |
Implements the Mini-Batch Gradient Descent optimization algorithm. |
ModifiedGradientDescentOptimizer<T> |
Modified Gradient Descent optimizer for Hope architecture. |
MomentumOptimizer<T, TInput, TOutput> |
Implements the Momentum optimization algorithm for gradient-based optimization. |
NadamOptimizer<T, TInput, TOutput> |
Implements the Nesterov-accelerated Adaptive Moment Estimation (Nadam) optimization algorithm. |
NelderMeadOptimizer<T, TInput, TOutput> |
Implements the Nelder-Mead optimization algorithm, also known as the downhill simplex method. |
NesterovAcceleratedGradientOptimizer<T, TInput, TOutput> |
Implements the Nesterov Accelerated Gradient optimization algorithm. |
NewtonMethodOptimizer<T, TInput, TOutput> |
Implements the Newton’s Method optimization algorithm. |
NormalOptimizer<T, TInput, TOutput> |
Implements a normal optimization algorithm with adaptive parameters. |
OptimizationDataBatcher<T, TInput, TOutput> |
Provides batch iteration utilities for optimization input data. |
ParticleSwarmOptimizer<T, TInput, TOutput> |
Implements a Particle Swarm Optimization algorithm for finding optimal solutions. |
ProximalGradientDescentOptimizer<T, TInput, TOutput> |
Implements a Proximal Gradient Descent optimization algorithm which combines gradient descent with regularization. |
RAdamOptimizer<T, TInput, TOutput> |
Implements RAdam (Rectified Adam), the variant of Adam that rectifies the variance of the adaptive learning rate instead of relying on a hand-tuned warmup schedule. |
RootMeanSquarePropagationOptimizer<T, TInput, TOutput> |
Implements the Root Mean Square Propagation (RMSProp) optimization algorithm, an adaptive learning rate method. |
RpropOptimizer<T, TInput, TOutput> |
Implements Rprop (resilient backpropagation): a per-weight adaptive step size driven by the SIGN of the gradient, with the magnitude discarded entirely. |
ScaledAdamOptimizer<T, TInput, TOutput> |
ScaledAdam (k2-fsa/icefall, as GPT-SoVITS vendors it in AR/modules/optim.py): Adam whose update to each tensor is scaled by that tensor’s root-mean-square, which also learns each tensor’s overall scale, with momentum kept as a parameter c… |
ScheduleFreeAdamWOptimizer<T, TInput, TOutput> |
Schedule-Free AdamW (Defazio et al. |
StochasticGradientDescentOptimizer<T, TInput, TOutput> |
Represents a Stochastic Gradient Descent (SGD) optimizer for machine learning models. |
TabuSearchOptimizer<T, TInput, TOutput> |
Represents a Tabu Search optimizer for machine learning models. |
TrustRegionOptimizer<T, TInput, TOutput> |
Implements the Trust Region optimization algorithm for machine learning models. |