RetNetLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the RetNet (Retentive Network) layer from Sun et al., 2023.

Inheritance: LayerBase<T> → RetNetLayer<T>

Implements: IShapeContract

For Beginners

RetNet is designed to be a successor to the Transformer architecture.

Think of a Transformer’s attention as: “For every word, look at ALL other words and decide what’s important.” This is powerful but expensive (quadratic cost).

RetNet replaces this with a “retention” mechanism that works like a fading memory:

  • Recent words are remembered clearly (high weight)
  • Older words gradually fade away (exponential decay)
  • Different heads “forget” at different speeds:
  • Some heads have long memory (gamma close to 1.0) - they capture long-range patterns
  • Some heads have short memory (gamma around 0.97) - they capture local patterns

The big advantage: RetNet can be computed three ways:

  1. Parallel mode (for training): process the whole sequence at once, like a Transformer
  2. Recurrent mode (for inference): process one token at a time, like an RNN - O(1) per step
  3. Chunkwise mode: a hybrid that balances speed and parallelism

This means RetNet trains like a Transformer but generates text like an RNN, getting the best of both worlds.

How It Works

RetNet introduces a multi-scale retention mechanism that replaces softmax attention with an exponential decay approach, supporting parallel, recurrent, and chunkwise computation modes. This implementation uses the parallel formulation for training:

where D is a causal decay mask with D_ij = gamma^(i-j) for i >= j and 0 otherwise.

The architecture:

The multi-scale retention is the key innovation: each head operates at a different time scale via its own decay rate gamma_h. Heads with gamma close to 1.0 retain long-range context, while heads with smaller gamma focus on local context. This naturally creates a multi-scale representation without requiring positional encodings.

The decay rates are initialized using the formula from the paper:

which spaces the decay rates logarithmically between approximately 0.97 and 0.9999, ensuring that heads cover a wide range of temporal scales.

Reference: Sun et al., “Retentive Network: A Successor to Transformer for Large Language Models”, 2023. https://arxiv.org/abs/2307.08621

Constructors

Constructor Summary
RetNetLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new RetNet (Retentive Network) layer.

Properties

Property Summary
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension (d_model).
int NumHeads { get; } Gets the number of retention heads.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns the complete convention-enumerated trainable tensor surface.
Tensor<T> GetDecayRates() Gets the per-head decay rates (gammas) for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters)
void UpdateParameters(T learningRate)