RootMeanSquarePropagationOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.Optimizers
Implements the Root Mean Square Propagation (RMSProp) optimization algorithm, an adaptive learning rate method.
Inheritance: OptimizerBase<T, TInput, TOutput> → GradientBasedOptimizerBase<T, TInput, TOutput> → RootMeanSquarePropagationOptimizer<T, TInput, TOutput>
For Beginners
RMSProp is like a hiker who adjusts their step size differently for each direction.
Imagine a hiker exploring mountains with different terrains:
- On steep slopes (large gradients), the hiker takes small, careful steps
- On gentle slopes (small gradients), the hiker takes larger, confident steps
- The hiker remembers how steep each direction has been recently (using a moving average)
- This memory helps the hiker adjust their steps even as the terrain changes
This adaptive approach helps the algorithm find good solutions more quickly by:
- Preventing wild overshooting on steep slopes
- Making faster progress on gentle terrain
- Adjusting automatically to different parts of the solution space
How It Works
RMSProp is an adaptive learning rate optimization algorithm designed to handle non-stationary objectives and accelerate convergence. It maintains a moving average of the squared gradients for each parameter and divides the learning rate by the square root of this average. This approach allows the algorithm to use a larger learning rate for parameters with small gradients and a smaller learning rate for parameters with large gradients, leading to more efficient optimization.
Example
using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.Enums;
using AiDotNet.NeuralNetworks;
using AiDotNet.Optimizers;
using AiDotNet.Tensors.LinearAlgebra;
var rng = new Random(0);
var trainX = new Tensor<double>(new[] { 32, 8 });
var trainY = new Tensor<double>(new[] { 32, 2 });
for (int i = 0; i < 32; i++)
{
for (int j = 0; j < 8; j++) trainX[new[] { i, j }] = rng.NextDouble();
trainY[new[] { i, i % 2 }] = 1.0;
}
var model = new NeuralNetwork<double>(new NeuralNetworkArchitecture<double>(
inputFeatures: 8, numClasses: 2, complexity: NetworkComplexity.Simple));
var result = await new AiModelBuilder<double, Tensor<double>, Tensor<double>>()
.ConfigureModel(model)
.ConfigureOptimizer(new RootMeanSquarePropagationOptimizer<double, Tensor<double>, Tensor<double>>(model))
.ConfigureDataLoader(DataLoaders.FromTensors(trainX, trainY))
.BuildAsync();
Console.WriteLine("Trained with RootMeanSquarePropagationOptimizer.");
Constructors
| Constructor | Summary |
|---|---|
RootMeanSquarePropagationOptimizer(IFullModel<T, TInput, TOutput> model, RootMeanSquarePropagationOptimizerOptions<T, TInput, TOutput> options, IEngine engine) |
Initializes a new instance of the RootMeanSquarePropagationOptimizer<T> class with the specified options and components. |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuUpdate { get; } |
Gets whether this optimizer supports GPU-accelerated parameter updates. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
Gets the current options for this optimizer. |
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
Performs the RMSProp optimization to find the best solution for the given input data. |
RootMeanSquarePropagationOptimizer<T, TInput, TOutput> CreateForFunction(RootMeanSquarePropagationOptimizerOptions<T, TInput, TOutput> options) |
Creates an RMSProp optimizer for minimizing a plain function, with no model attached. |
Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients) |
Reverses an RMSprop gradient update to recover original parameters. |
Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient) |
Updates a vector of parameters using the RMSProp algorithm. |
void DisposeGpuState() |
Disposes GPU-allocated optimizer state. |
void InitializeGpuState(int parameterCount, IDirectGpuBackend backend) |
Initializes RMSprop optimizer state on the GPU. |
void Reset() |
Resets the optimizer to its initial state. |
void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend) |
Updates parameters on the GPU using the RMSprop kernel. |