TTTLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the TTT (Test-Time Training) layer from Sun et al., 2024.
Inheritance: LayerBase<T> → TTTLayer<T>
Implements: IShapeContract
For Beginners
Traditional RNNs store their memory as a fixed-size vector (like a notepad with limited space). TTT stores memory as a small neural network’s weights (like having a student who learns from each example).
At each step in the sequence:
- The “student” (inner model W) tries to predict the value from the key
- It computes how wrong it was (the loss)
- It takes a learning step to improve (gradient descent)
- Then it answers a query using its updated knowledge
This means TTT can adapt to the specific patterns in each sequence, much like how you get better at a task the more examples you see. The inner learning rate (eta) controls how quickly the student adapts – too fast and it forgets old information, too slow and it cannot keep up.
TTT-Linear is competitive with Transformers and Mamba on language modeling benchmarks while maintaining linear O(n) complexity in sequence length.
How It Works
TTT replaces the fixed-size hidden state of traditional RNNs with a more expressive hidden state: the WEIGHTS of a small inner model. At each timestep the layer performs a gradient-based learning step on those weights, effectively training the inner model on the fly while processing the sequence.
This file implements the TTT-Linear variant where the inner model is a single linear map W. The recurrence at each timestep t is:
The key insight is that the hidden state (W) grows in expressivity with the size of the inner model. Unlike a fixed-size RNN state vector, W can represent arbitrary linear relationships and is updated using a principled gradient descent rule (the delta rule). This makes TTT a bridge between recurrent models and in-context learning: the model literally “learns at test time.”
Multi-head operation: each head maintains its own inner weight matrix W_h of size [headDim, headDim]. This allows different heads to learn different relationships from the sequence, similar to multi-head attention. The heads operate independently and their outputs are concatenated.
Reference: Sun et al., “Learning to (Learn at Test Time): RNNs with Expressive Hidden States”, 2024.
https://arxiv.org/abs/2407.04620
Constructors
| Constructor | Summary |
|---|---|
TTTLayer(int sequenceLength, int modelDimension, int numHeads, double innerLearningRate, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new TTT (Test-Time Training) layer implementing the TTT-Linear variant. |
Properties
| Property | Summary |
|---|---|
T InnerLearningRate { get; } |
Gets the inner learning rate used for test-time training updates. |
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetInnerWeightsInit() |
Gets the inner model initial weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |