ModifiedGradientDescentOptimizer<T>
Models & Types · AiDotNet.Optimizers
Modified Gradient Descent optimizer for Hope architecture. Based on Equations 27-29 from “Nested Learning” paper.
Traditional GD: W_{t+1} = W_t - η * ∇L(W_t; x_t) ⊗ x_t Modified GD: W_{t+1} = W_t * (I - x_t*x_t^T) - η * ∇L(W_t; x_t) ⊗ x_t
This formulation uses L2 regression objective instead of dot-product similarity, resulting in better handling of data dependencies in token space.
Constructors
| Constructor | Summary |
|---|---|
ModifiedGradientDescentOptimizer(T learningRate, IEngine engine) |
Creates a modified gradient descent optimizer. |
Properties
| Property | Summary |
|---|---|
T LearningRate { get; } |
Gets the learning rate. |
Methods
| Method | Summary |
|---|---|
Matrix<T> UpdateMatrix(Matrix<T> currentParameters, Vector<T> input, Vector<T> outputGradient) |
Updates parameters using modified gradient descent (Equations 27-29). |
Vector<T> UpdateVector(Vector<T> currentParameters, Vector<T> input, Vector<T> outputGradient) |
Updates a parameter vector using modified gradient descent. |