MesaNetLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the MesaNet layer from Grazzi et al., 2025.
Inheritance: LayerBase<T> → MesaNetLayer<T>
Implements: IShapeContract
For Beginners
Imagine you have a student (the inner model W) who learns from examples.
TTT approach: After each example, the student takes a small step toward the right answer (gradient descent). The step size (learning rate) is tricky to set – too big and the student overshoots, too small and learning is slow.
MesaNet approach: After each example, the student computes the BEST POSSIBLE answer given all examples seen so far, with a gentle preference toward their initial knowledge (regularization). There is no step size to tune – the answer is mathematically optimal.
The Woodbury trick makes this efficient: instead of re-solving the entire problem from scratch each time, MesaNet incrementally updates the solution as new examples arrive. This is like updating a running average instead of recomputing from all data points.
How It Works
MesaNet (“Locally Optimal Test-Time Training”) improves upon TTT by replacing gradient descent with a closed-form ridge regression update for the inner model weights. Instead of taking a noisy gradient step at each timestep, MesaNet computes the locally optimal weight matrix W_t that minimizes the reconstruction error plus a regularization term toward the initial weights W_0.
The optimization at each timestep t:
The Woodbury identity allows rank-1 updates to the inverse covariance matrix P, making each step O(d^2) instead of O(d^3) for a full matrix inversion. This keeps the overall complexity linear in sequence length while achieving a strictly better update than gradient descent.
Why this is better than TTT’s gradient descent:
- TTT: W_t = W_{t-1} - eta * grad (approximate, depends on learning rate choice)
- MesaNet: W_t = optimal solution of ridge regression (exact, only depends on lambda)
- No learning rate sensitivity – lambda is more stable and interpretable
- Converges in one step per observation rather than requiring multiple gradient steps
Reference: Grazzi et al., “MesaNet: Locally Optimal Test-Time Training”, 2025.
https://arxiv.org/abs/2506.05233
Constructors
| Constructor | Summary |
|---|---|
MesaNetLayer(int sequenceLength, int modelDimension, int numHeads, double regularization, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new MesaNet layer implementing locally optimal test-time training. |
Properties
| Property | Summary |
|---|---|
T Regularization { get; } |
Gets the regularization strength (lambda). |
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetInnerWeightsInit() |
Gets the inner model initial weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |