GatedDeltaProductLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Gated DeltaProduct layer from “DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders” (Siems et al., 2025).
Inheritance: LayerBase<T> → GatedDeltaProductLayer<T>
Implements: IShapeContract
For Beginners
Gated DeltaProduct is the fully-featured version that combines everything from both GatedDeltaNet and DeltaProduct.
Think of managing a library:
- Alpha (forget gate): “Remove 10% of books from each shelf” (uniform fading)
- Householder product: “Rearrange remaining books by topic instead of author” (rotation)
- Beta (write gate): “How many new books to add to the collection” (write strength)
- Output gate: “Which shelves to show to the current visitor” (output filtering)
Without gating (plain DeltaProduct): you can rearrange and add books, but can’t thin out the collection. Without Householder (plain GatedDeltaNet): you can thin out and add, but can’t rearrange. Gated DeltaProduct does all four operations at each step.
This makes it the most expressive variant in the DeltaNet family, at the cost of slightly more compute per step.
How It Works
Gated DeltaProduct combines the Householder product state transitions of DeltaProduct with output gating (similar to how GatedDeltaNet adds gating to DeltaNet). The forget gate (alpha) provides an additional scalar decay before the Householder rotation, and the output gate controls information flow to the final output.
The architecture:
The combination of gating and Householder products provides maximum expressivity:
- Alpha gate: Controls how much old state to retain (like GatedDeltaNet)
- Householder product: ROTATES the retained state (like DeltaProduct)
- Beta gate: Controls how strongly to write new information
- Output gate: Controls what to expose from the state
This means the model can: fade old information (alpha), rearrange what remains (Householder), write new information selectively (beta), and filter what to output (gate).
Reference: Siems et al., “DeltaProduct: Increasing the Expressivity of DeltaNet Through
Products of Householders”, 2025. https://arxiv.org/abs/2502.10297
Constructors
| Constructor | Summary |
|---|---|
GatedDeltaProductLayer(int sequenceLength, int modelDimension, int numHeads, int numHouseholders, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Gated DeltaProduct layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
int NumHouseholders { get; } |
Gets the number of Householder reflections per timestep. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetHouseholderWeights() |
Gets the Householder projection weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |