HGRNLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Hierarchically Gated Recurrent Neural Network (HGRN) layer from NeurIPS 2023.
Inheritance: LayerBase<T> → HGRNLayer<T>
Implements: IShapeContract
For Beginners
HGRN is like a stack of simple “memory filters” that work at different speeds.
Imagine you’re listening to music:
- One filter (lower layer, high forget bias) remembers the overall melody for a long time
– it barely forgets anything, holding onto the big picture
- Another filter (upper layer, low forget bias) tracks the current beat and rhythm
– it quickly forgets old beats to focus on what’s happening right now
- Together, they understand both the long melody AND the short rhythm simultaneously
Each filter is extremely simple: at every step, it just does: new_memory = (forget_amount * old_memory) + (input_amount * new_input)
This is much simpler than Transformers (which compare every position to every other position) or even Mamba (which maintains matrix-valued states). Yet by stacking these simple filters with different forgetting speeds, HGRN achieves competitive performance with linear O(n) complexity – it processes a sequence of length n in time proportional to n, not n^2.
How It Works
HGRN uses hierarchical gating to achieve multi-scale temporal processing with linear O(n) complexity. Each layer performs a simple element-wise gated recurrence:
The “hierarchical” aspect is the key insight: when stacking multiple HGRN layers, each layer uses a different forget gate bias. Lower layers use higher forget gate bias values (sigmoid output closer to 1), creating slow-decaying memory that captures long-range dependencies. Upper layers use lower bias values (sigmoid output closer to 0), creating fast-decaying memory for short-range local patterns. This naturally creates a multi-scale temporal hierarchy where different layers specialize in different time scales, similar to how wavelets decompose signals at multiple resolutions.
Unlike GatedDeltaNet or Mamba which maintain matrix-valued states, HGRN maintains only a vector-valued hidden state h_t (same dimension as the input). This makes it extremely memory-efficient and fast, while the hierarchical gating compensates for the simpler state by distributing temporal modeling across layers.
Reference: Qin et al., “Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, NeurIPS 2023.
https://arxiv.org/abs/2311.04823
Constructors
| Constructor | Summary |
|---|---|
HGRNLayer(int sequenceLength, int modelDimension, double forgetBias, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new HGRN (Hierarchically Gated Recurrent Neural Network) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
double ForgetBias { get; } |
Gets the forget gate bias value used for hierarchical gating. |
int ModelDimension { get; } |
Gets the model dimension (input/output width). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetForgetGateWeights() |
Gets the forget gate weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |