GatedDeltaNetLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the GatedDeltaNet layer from NVIDIA, ICLR 2025.

Inheritance: LayerBase<T> → GatedDeltaNetLayer<T>

Implements: IShapeContract

For Beginners

GatedDeltaNet is one of the best sub-quadratic architectures as of 2025.

Think of the state matrix S as a “lookup table” that maps keys to values:

  • Linear attention: “Just add key-value pairs to the table” -> entries pile up, old ones never corrected
  • Delta rule: “Before adding, check if this key already has a value. Only write the correction.”

This is like the difference between:

  • Memorizing every flashcard answer independently (linear attention)
  • Checking what you already know, then only memorizing what’s new or different (delta rule)

The gating mechanism (alpha, beta) lets the model control:

  • How much to forget old entries (alpha)
  • How strongly to write new corrections (beta)

Combined with a short convolution for local context, this simple recipe matches Transformers while being much more efficient for long sequences.

How It Works

GatedDeltaNet combines the delta rule for fast weight updates with gated output, achieving state-of-the-art performance among sub-quadratic architectures. It matches Transformer quality on many benchmarks while maintaining linear O(n) complexity.

The architecture:

The delta rule update is key: instead of blindly accumulating KV outer products (like linear attention), it computes the error (V - SK) first and updates accordingly. This is exactly the delta rule from neural network learning theory, applied to the fast weight matrix at each step.

Reference: Yang et al., “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR 2025. https://arxiv.org/abs/2412.06464

Constructors

Constructor Summary
GatedDeltaNetLayer(int sequenceLength, int modelDimension, int numHeads, int convKernelSize, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new GatedDeltaNet layer.

Properties

Property Summary
bool SupportsTraining { get; }
int ConvKernelSize { get; } Gets the convolution kernel size.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).