RealGatedLinearRecurrenceLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Real-Gated Linear Recurrence Unit (RG-LRU) from Google DeepMind’s Griffin architecture.

Inheritance: LayerBase<T> → RealGatedLinearRecurrenceLayer<T>

Implements: IShapeContract

For Beginners

The RG-LRU is like a learnable “leaky bucket” for information.

Imagine each position in your hidden state as a bucket:

  • The recurrence gate (r) controls how much water leaks out each step (memory decay)
  • The input gate (i) controls how much new water pours in
  • The sqrt(1 - a^2) factor ensures the bucket never overflows or runs dry

This is simpler than Mamba (no Conv1D, no SSM parameters B/C) but surprisingly effective. Google’s RecurrentGemma models (2B, 9B) use this architecture and achieve competitive performance with Transformer-based Gemma models.

How It Works

The RG-LRU is a gated linear recurrence that serves as the core sequence mixing mechanism in the Griffin and Hawk architectures. It uses input-dependent gating to control both the recurrence decay and the input contribution, providing selective memory similar to Mamba but through a different mathematical formulation.

The recurrence is:

The sqrt(1 - a_t^2) factor ensures the recurrence preserves signal magnitude, preventing vanishing or exploding states.

Griffin combines RG-LRU with local attention in a hybrid architecture. This layer implements the RG-LRU component which can be used standalone or as part of a hybrid.

Reference: De et al., “Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models”, 2024. https://arxiv.org/abs/2402.19427

Constructors

Constructor Summary
RealGatedLinearRecurrenceLayer(int sequenceLength, int modelDimension, int recurrenceDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Real-Gated Linear Recurrence Unit (RG-LRU) layer.

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (input/output width).
int RecurrenceDimension { get; } Gets the recurrence dimension (hidden state width).

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns the complete convention-enumerated trainable tensor surface.
Tensor<T> GetDecayParameter() Gets the decay parameter for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters)
void UpdateParameters(T learningRate)