RetNetLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the RetNet (Retentive Network) layer from Sun et al., 2023.
Inheritance: LayerBase<T> → RetNetLayer<T>
Implements: IShapeContract
For Beginners
RetNet is designed to be a successor to the Transformer architecture.
Think of a Transformer’s attention as: “For every word, look at ALL other words and decide what’s important.” This is powerful but expensive (quadratic cost).
RetNet replaces this with a “retention” mechanism that works like a fading memory:
- Recent words are remembered clearly (high weight)
- Older words gradually fade away (exponential decay)
- Different heads “forget” at different speeds:
- Some heads have long memory (gamma close to 1.0) - they capture long-range patterns
- Some heads have short memory (gamma around 0.97) - they capture local patterns
The big advantage: RetNet can be computed three ways:
- Parallel mode (for training): process the whole sequence at once, like a Transformer
- Recurrent mode (for inference): process one token at a time, like an RNN - O(1) per step
- Chunkwise mode: a hybrid that balances speed and parallelism
This means RetNet trains like a Transformer but generates text like an RNN, getting the best of both worlds.
How It Works
RetNet introduces a multi-scale retention mechanism that replaces softmax attention with an exponential decay approach, supporting parallel, recurrent, and chunkwise computation modes. This implementation uses the parallel formulation for training:
where D is a causal decay mask with D_ij = gamma^(i-j) for i >= j and 0 otherwise.
The architecture:
The multi-scale retention is the key innovation: each head operates at a different time scale via its own decay rate gamma_h. Heads with gamma close to 1.0 retain long-range context, while heads with smaller gamma focus on local context. This naturally creates a multi-scale representation without requiring positional encodings.
The decay rates are initialized using the formula from the paper:
which spaces the decay rates logarithmically between approximately 0.97 and 0.9999, ensuring that heads cover a wide range of temporal scales.
Reference: Sun et al., “Retentive Network: A Successor to Transformer for Large Language Models”, 2023.
https://arxiv.org/abs/2307.08621
Constructors
| Constructor | Summary |
|---|---|
RetNetLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new RetNet (Retentive Network) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumHeads { get; } |
Gets the number of retention heads. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetDecayRates() |
Gets the per-head decay rates (gammas) for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |