ModifiedGradientDescentOptimizer<T>

Models & Types · AiDotNet.Optimizers

Modified Gradient Descent optimizer for Hope architecture. Based on Equations 27-29 from “Nested Learning” paper.

Traditional GD: W_{t+1} = W_t - η * ∇L(W_t; x_t) ⊗ x_t Modified GD: W_{t+1} = W_t * (I - x_t*x_t^T) - η * ∇L(W_t; x_t) ⊗ x_t

This formulation uses L2 regression objective instead of dot-product similarity, resulting in better handling of data dependencies in token space.

Constructors

Constructor Summary
ModifiedGradientDescentOptimizer(T learningRate, IEngine engine) Creates a modified gradient descent optimizer.

Properties

Property Summary
T LearningRate { get; } Gets the learning rate.

Methods

Method Summary
Matrix<T> UpdateMatrix(Matrix<T> currentParameters, Vector<T> input, Vector<T> outputGradient) Updates parameters using modified gradient descent (Equations 27-29).
Vector<T> UpdateVector(Vector<T> currentParameters, Vector<T> input, Vector<T> outputGradient) Updates a parameter vector using modified gradient descent.