DeltaNetLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the DeltaNet layer from “Linear Transformers with Learnable Kernel Functions” (Yang et al., 2024).

Inheritance: LayerBase<T> → DeltaNetLayer<T>

Implements: IShapeContract

For Beginners

DeltaNet is a simpler, foundational variant of GatedDeltaNet.

Think of the state matrix S as a “lookup table” that maps keys to values:

  • Linear attention: “Just add every key-value pair to the table” -> entries pile up, old ones never corrected
  • Delta rule: “Before adding, check what S already predicts for this key. Only write the correction.”

This is like the difference between:

  • Writing every flashcard answer on top of the previous one (linear attention -> messy)
  • Erasing only the wrong part and writing the correction (delta rule -> clean)

The beta parameter controls how much of the correction to actually apply:

  • beta near 0: “I trust the existing memory, barely update”
  • beta near 1: “Fully overwrite whatever was stored for this key”

Because there is no forget gate (alpha) or output gate, this model is simpler and faster than GatedDeltaNet, but may underperform on tasks that require selective forgetting or output gating.

How It Works

DeltaNet applies the delta rule to linear attention, replacing the naive accumulation of key-value outer products with an error-corrective update. This produces a linear-complexity recurrent model that is significantly more expressive than standard linear attention.

The architecture:

This is the “ungated” version of GatedDeltaNet: there is no alpha forget gate, no output gate, and no short convolution. The state is purely additive (S_{t-1} carries forward with weight 1), and the only learned control is beta which modulates how strongly corrections are written.

The delta rule update is key: instead of blindly accumulating KV outer products (like linear attention), it computes the error (V - SK) first and updates accordingly. This is exactly the Widrow-Hoff / delta rule from neural network learning theory, applied to a fast weight matrix at each timestep.

Reference: Yang et al., “Linear Transformers with Learnable Kernel Functions”, 2024. https://arxiv.org/abs/2406.06484

Constructors

Constructor Summary
DeltaNetLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new DeltaNet layer.

Properties

Property Summary
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head (modelDimension / numHeads).
int ModelDimension { get; } Gets the model dimension (d_model).
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection or analysis.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection or analysis.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).