RealGatedLinearRecurrenceLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Real-Gated Linear Recurrence Unit (RG-LRU) from Google DeepMind’s Griffin architecture.
Inheritance: LayerBase<T> → RealGatedLinearRecurrenceLayer<T>
Implements: IShapeContract
For Beginners
The RG-LRU is like a learnable “leaky bucket” for information.
Imagine each position in your hidden state as a bucket:
- The recurrence gate (r) controls how much water leaks out each step (memory decay)
- The input gate (i) controls how much new water pours in
- The sqrt(1 - a^2) factor ensures the bucket never overflows or runs dry
This is simpler than Mamba (no Conv1D, no SSM parameters B/C) but surprisingly effective. Google’s RecurrentGemma models (2B, 9B) use this architecture and achieve competitive performance with Transformer-based Gemma models.
How It Works
The RG-LRU is a gated linear recurrence that serves as the core sequence mixing mechanism in the Griffin and Hawk architectures. It uses input-dependent gating to control both the recurrence decay and the input contribution, providing selective memory similar to Mamba but through a different mathematical formulation.
The recurrence is:
The sqrt(1 - a_t^2) factor ensures the recurrence preserves signal magnitude, preventing vanishing or exploding states.
Griffin combines RG-LRU with local attention in a hybrid architecture. This layer implements the RG-LRU component which can be used standalone or as part of a hybrid.
Reference: De et al., “Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models”, 2024.
https://arxiv.org/abs/2402.19427
Constructors
| Constructor | Summary |
|---|---|
RealGatedLinearRecurrenceLayer(int sequenceLength, int modelDimension, int recurrenceDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Real-Gated Linear Recurrence Unit (RG-LRU) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (input/output width). |
int RecurrenceDimension { get; } |
Gets the recurrence dimension (hidden state width). |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetDecayParameter() |
Gets the decay parameter for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |