OptimizerType

Enums · AiDotNet.Enums

Defines different optimization algorithms used to train machine learning models.

For Beginners

Optimizers are like the “learning strategy” for AI models. When an AI model is learning, it needs to find the best values for its internal settings (called parameters or weights). Optimizers are different methods for finding these best values efficiently. Think of them as different strategies for climbing a mountain to reach the peak - some take small careful steps, others take bigger leaps, and some use special techniques to avoid getting stuck on small hills before reaching the highest peak.

Fields

Field Summary
OptimizerType ADMM Alternating Direction Method of Multipliers — splits a problem into pieces solved in turn.
OptimizerType AMSGrad Variant of Adam that maintains the maximum of past squared gradients for better convergence.
OptimizerType ASGD Averaged Stochastic Gradient Descent: decayed SGD that also tracks a running average of the iterates.
OptimizerType AdaDelta Extension of Adagrad that uses a moving average of squared gradients to adapt the learning rate.
OptimizerType AdaGrad Adaptive gradient algorithm that accumulates squared gradients to scale the learning rate.
OptimizerType AdaMax Variant of Adam that uses the infinity norm for more stable updates.
OptimizerType Adadelta Extension of Adagrad that uses a different approach to address the diminishing learning rates problem.
OptimizerType Adagrad Adaptive learning rate method that scales learning rates individually for each parameter.
OptimizerType Adam Adaptive Moment Estimation - combines the benefits of momentum and RMSProp optimizers.
OptimizerType Adam8Bit Adam with 8-bit quantized optimizer state.
OptimizerType AdamOptimizer Combination of adaptive learning rates and momentum for efficient optimization.
OptimizerType AdamW Variant of Adam that implements a decoupled weight decay for better generalization.
OptimizerType Adamax Variant of Adam that uses the infinity norm instead of the L2 norm.
OptimizerType AdaptiveGradient Adaptive gradient algorithm that maintains per-parameter learning rates.
OptimizerType AntColony Nature-inspired algorithm based on the foraging behavior of ant colonies.
OptimizerType BFGS Broyden-Fletcher-Goldfarb-Shanno — a quasi-Newton method that builds an approximate inverse Hessian.
OptimizerType BayesianOptimization Optimization approach that builds a probabilistic model of the objective function.
OptimizerType CMAES Covariance Matrix Adaptation Evolution Strategy — a derivative-free evolutionary optimizer.
OptimizerType ConjugateGradient Optimization method that generates search directions that don’t interfere with previous progress.
OptimizerType CoordinateDescent Optimization method that updates one parameter at a time while keeping others fixed.
OptimizerType CrossEntropy Optimization method that generates random samples and selects the best performing ones.
OptimizerType DFP Davidon-Fletcher-Powell — the earlier quasi-Newton update formula that BFGS refined.
OptimizerType DifferentialEvolution Evolutionary algorithm that creates new solutions by combining existing ones.
OptimizerType EvolutionaryAlgorithm Evolutionary algorithm that uses principles of natural selection to optimize solutions.
OptimizerType FTRL Online learning algorithm that adapts to the data characteristics.
OptimizerType GeneticAlgorithm Evolutionary algorithm inspired by natural selection and genetics.
OptimizerType GradientDescent Classic optimization algorithm that updates parameters in the direction of steepest descent.
OptimizerType HillClimbing Simple optimization strategy that always moves in the direction of immediate improvement.
OptimizerType LAMB Layer-wise Adaptive Moments optimizer for Batch training — Adam with a per-layer trust ratio.
OptimizerType LARS Layer-wise Adaptive Rate Scaling — SGD with momentum plus a per-layer trust ratio.
OptimizerType LBFGS Second-order optimization method that approximates the Hessian matrix to accelerate convergence.
OptimizerType LevenbergMarquardt Levenberg-Marquardt — interpolates between gradient descent and Gauss-Newton for least-squares fits.
OptimizerType Lion Optimization algorithm that combines the Lion optimizer with advanced activation techniques.
OptimizerType MiniBatchGradientDescent Mini-batch gradient descent — plain gradient descent over fixed-size batches.
OptimizerType Momentum Gradient descent variant that adds a fraction of the previous update to the current one.
OptimizerType Nadam Combination of Adam and Nesterov momentum for improved performance.
OptimizerType NelderMead Direct search method that doesn’t require gradient information.
OptimizerType NestedLearning Nested Learning optimizer - a multi-level optimization paradigm for continual learning.
OptimizerType NesterovAcceleratedGradient Advanced variant of Momentum that looks ahead to where the current momentum would take it.
OptimizerType NewtonMethod Newton’s method — uses the exact second derivative (Hessian) rather than an approximation.
OptimizerType Normal Standard optimization approach with fixed learning rate.
OptimizerType NormalOptimizer Optimizer that combines random search with adaptive parameter tuning.
OptimizerType ParticleSwarm Swarm intelligence algorithm inspired by the social behavior of birds or fish.
OptimizerType PowellMethod Optimization method that performs sequential line minimizations along conjugate directions.
OptimizerType ProximalGradientDescent Proximal gradient descent — a gradient step followed by a proximal operator, for non-smooth penalties.
OptimizerType QuasiNewton Optimization method that approximates second-order information for better search directions.
OptimizerType RAdam Variant of Adam that rectifies the variance of the adaptive learning rate.
OptimizerType RMSProp Extension of Adagrad that addresses the diminishing learning rates problem.
OptimizerType Rprop Resilient backpropagation: adapts a per-weight step size from the sign of the gradient, ignoring its size.
OptimizerType SimulatedAnnealing Probabilistic technique inspired by the annealing process in metallurgy.
OptimizerType StochasticGradientDescent Variant of gradient descent that uses random subsets of data for each update.
OptimizerType TabuSearch Tabu search — local search that keeps a short-term memory of recent moves and forbids repeating them.
OptimizerType TrustRegion Optimization method that uses a simplified model within a trusted region.
int value__