HGRNLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Hierarchically Gated Recurrent Neural Network (HGRN) layer from NeurIPS 2023.

Inheritance: LayerBase<T> → HGRNLayer<T>

Implements: IShapeContract

For Beginners

HGRN is like a stack of simple “memory filters” that work at different speeds.

Imagine you’re listening to music:

  • One filter (lower layer, high forget bias) remembers the overall melody for a long time

– it barely forgets anything, holding onto the big picture

  • Another filter (upper layer, low forget bias) tracks the current beat and rhythm

– it quickly forgets old beats to focus on what’s happening right now

  • Together, they understand both the long melody AND the short rhythm simultaneously

Each filter is extremely simple: at every step, it just does: new_memory = (forget_amount * old_memory) + (input_amount * new_input)

This is much simpler than Transformers (which compare every position to every other position) or even Mamba (which maintains matrix-valued states). Yet by stacking these simple filters with different forgetting speeds, HGRN achieves competitive performance with linear O(n) complexity – it processes a sequence of length n in time proportional to n, not n^2.

How It Works

HGRN uses hierarchical gating to achieve multi-scale temporal processing with linear O(n) complexity. Each layer performs a simple element-wise gated recurrence:

The “hierarchical” aspect is the key insight: when stacking multiple HGRN layers, each layer uses a different forget gate bias. Lower layers use higher forget gate bias values (sigmoid output closer to 1), creating slow-decaying memory that captures long-range dependencies. Upper layers use lower bias values (sigmoid output closer to 0), creating fast-decaying memory for short-range local patterns. This naturally creates a multi-scale temporal hierarchy where different layers specialize in different time scales, similar to how wavelets decompose signals at multiple resolutions.

Unlike GatedDeltaNet or Mamba which maintain matrix-valued states, HGRN maintains only a vector-valued hidden state h_t (same dimension as the input). This makes it extremely memory-efficient and fast, while the hierarchical gating compensates for the simpler state by distributing temporal modeling across layers.

Reference: Qin et al., “Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, NeurIPS 2023. https://arxiv.org/abs/2311.04823

Constructors

Constructor Summary
HGRNLayer(int sequenceLength, int modelDimension, double forgetBias, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new HGRN (Hierarchically Gated Recurrent Neural Network) layer.

Properties

Property Summary
bool SupportsTraining { get; }
double ForgetBias { get; } Gets the forget gate bias value used for hierarchical gating.
int ModelDimension { get; } Gets the model dimension (input/output width).

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetForgetGateWeights() Gets the forget gate weights for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).