MesaNetLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the MesaNet layer from Grazzi et al., 2025.

Inheritance: LayerBase<T> → MesaNetLayer<T>

Implements: IShapeContract

For Beginners

Imagine you have a student (the inner model W) who learns from examples.

TTT approach: After each example, the student takes a small step toward the right answer (gradient descent). The step size (learning rate) is tricky to set – too big and the student overshoots, too small and learning is slow.

MesaNet approach: After each example, the student computes the BEST POSSIBLE answer given all examples seen so far, with a gentle preference toward their initial knowledge (regularization). There is no step size to tune – the answer is mathematically optimal.

The Woodbury trick makes this efficient: instead of re-solving the entire problem from scratch each time, MesaNet incrementally updates the solution as new examples arrive. This is like updating a running average instead of recomputing from all data points.

How It Works

MesaNet (“Locally Optimal Test-Time Training”) improves upon TTT by replacing gradient descent with a closed-form ridge regression update for the inner model weights. Instead of taking a noisy gradient step at each timestep, MesaNet computes the locally optimal weight matrix W_t that minimizes the reconstruction error plus a regularization term toward the initial weights W_0.

The optimization at each timestep t:

The Woodbury identity allows rank-1 updates to the inverse covariance matrix P, making each step O(d^2) instead of O(d^3) for a full matrix inversion. This keeps the overall complexity linear in sequence length while achieving a strictly better update than gradient descent.

Why this is better than TTT’s gradient descent:

  • TTT: W_t = W_{t-1} - eta * grad (approximate, depends on learning rate choice)
  • MesaNet: W_t = optimal solution of ridge regression (exact, only depends on lambda)
  • No learning rate sensitivity – lambda is more stable and interpretable
  • Converges in one step per observation rather than requiring multiple gradient steps

Reference: Grazzi et al., “MesaNet: Locally Optimal Test-Time Training”, 2025. https://arxiv.org/abs/2506.05233

Constructors

Constructor Summary
MesaNetLayer(int sequenceLength, int modelDimension, int numHeads, double regularization, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new MesaNet layer implementing locally optimal test-time training.

Properties

Property Summary
T Regularization { get; } Gets the regularization strength (lambda).
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetInnerWeightsInit() Gets the inner model initial weights for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).