RootMeanSquarePropagationOptimizerOptions<T, TInput, TOutput>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the Root Mean Square Propagation (RMSProp) optimizer, an adaptive learning rate optimization algorithm commonly used in training neural networks.
Inheritance: ModelOptions → OptimizationAlgorithmOptions<T, TInput, TOutput> → GradientBasedOptimizerOptions<T, TInput, TOutput> → RootMeanSquarePropagationOptimizerOptions<T, TInput, TOutput>
For Beginners
RMSProp is an optimization algorithm that helps neural networks learn more efficiently.
When training a neural network or other machine learning model:
- We need to adjust the model’s parameters to minimize errors
- Different parameters may need different adjustment rates
- Some directions in the parameter space may need larger or smaller steps
RMSProp solves these problems by:
- Tracking the recent history of gradients (how parameters should change)
- Automatically adjusting the learning rate for each parameter
- Making larger updates for parameters with small or infrequent gradients
- Making smaller updates for parameters with large or frequent gradients
This adaptive behavior helps the model:
- Learn faster overall
- Avoid getting stuck in poor solutions
- Handle different types of features more effectively
RMSProp is particularly good for:
- Deep neural networks
- Recurrent neural networks
- Problems where different parameters need different learning rates
This class lets you configure the specific behavior of the RMSProp optimizer.
How It Works
RMSProp (Root Mean Square Propagation) is an adaptive learning rate optimization algorithm designed to address the diminishing learning rates problem of AdaGrad. Proposed by Geoffrey Hinton, RMSProp divides the learning rate for each parameter by a running average of the magnitudes of recent gradients for that parameter. Unlike AdaGrad, which accumulates all past squared gradients, RMSProp uses an exponentially decaying average, which prevents the learning rate from becoming infinitesimally small over time. This makes RMSProp particularly well-suited for non-stationary objectives and problems with noisy gradients. This class extends GradientBasedOptimizerOptions to provide specific configuration parameters for the RMSProp algorithm, including the decay rate for the moving average and a small epsilon value to prevent division by zero.
Constructors
| Constructor | Summary |
|---|---|
RootMeanSquarePropagationOptimizerOptions() |
Properties
| Property | Summary |
|---|---|
bool Centered { get; set; } |
Gets or sets whether RMSProp normalizes by an estimate of gradient variance, E[g²] - E[g]², rather than the uncentered second moment. |
bool EpsilonInsideRoot { get; set; } |
Gets or sets whether ε is added inside the square root, g/√(s + ε) (TensorFlow), rather than outside it, g/(√s + ε) (PyTorch, the default). |
double Decay { get; set; } |
Gets or sets the decay rate for the moving average of squared gradients. |
double Epsilon { get; set; } |
Gets or sets a small constant added to the denominator to improve numerical stability. |
double InitialMeanSquare { get; set; } |
The value every parameter’s running mean of squared gradients starts at. |
double Momentum { get; set; } |
Gets or sets the momentum of the uncentered update: v ← μ·v + η·g/√(s + ε), θ ← θ − v. |
int BatchSize { get; set; } |
Gets or sets the batch size for mini-batch gradient descent. |