GatedDeltaNetLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the GatedDeltaNet layer from NVIDIA, ICLR 2025.
Inheritance: LayerBase<T> → GatedDeltaNetLayer<T>
Implements: IShapeContract
For Beginners
GatedDeltaNet is one of the best sub-quadratic architectures as of 2025.
Think of the state matrix S as a “lookup table” that maps keys to values:
- Linear attention: “Just add key-value pairs to the table” -> entries pile up, old ones never corrected
- Delta rule: “Before adding, check if this key already has a value. Only write the correction.”
This is like the difference between:
- Memorizing every flashcard answer independently (linear attention)
- Checking what you already know, then only memorizing what’s new or different (delta rule)
The gating mechanism (alpha, beta) lets the model control:
- How much to forget old entries (alpha)
- How strongly to write new corrections (beta)
Combined with a short convolution for local context, this simple recipe matches Transformers while being much more efficient for long sequences.
How It Works
GatedDeltaNet combines the delta rule for fast weight updates with gated output, achieving state-of-the-art performance among sub-quadratic architectures. It matches Transformer quality on many benchmarks while maintaining linear O(n) complexity.
The architecture:
The delta rule update is key: instead of blindly accumulating KV outer products (like linear attention), it computes the error (V - SK) first and updates accordingly. This is exactly the delta rule from neural network learning theory, applied to the fast weight matrix at each step.
Reference: Yang et al., “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR 2025.
https://arxiv.org/abs/2412.06464
Constructors
| Constructor | Summary |
|---|---|
GatedDeltaNetLayer(int sequenceLength, int modelDimension, int numHeads, int convKernelSize, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new GatedDeltaNet layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ConvKernelSize { get; } |
Gets the convolution kernel size. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |