S4DLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements a Diagonal State Space (S4D) layer from Gu et al., 2022.

Inheritance: LayerBase<T> → S4DLayer<T>

Implements: IShapeContract

For Beginners

Think of S4D as a set of independent oscillators, each tuned to a different frequency. When you feed in a sequence, each oscillator responds to the parts of the signal at its frequency, building up a rich representation of the input over time.

The key insight is that diagonal state matrices are much simpler than full matrices:

  • Full S4: A is N x N -> complex eigenvalue decomposition needed
  • S4D: A is diagonal -> each state dimension is just one number

This makes S4D much faster while being nearly as expressive. It’s the foundation that led to more advanced models like Mamba.

How It Works

S4D simplifies the original S4 model by using a diagonal state matrix A, which greatly reduces computational complexity while maintaining competitive performance. The diagonal structure means each state dimension evolves independently, enabling efficient parallelization.

The layer implements the continuous-time state space model:

where A is diagonal (and typically complex-valued, stored as real/imaginary pairs for generic compatibility). Discretization uses the Zero-Order Hold (ZOH) method:

During training, the layer supports a global convolution mode that convolves the entire sequence at once using the closed-form convolution kernel. During inference, it uses an efficient recurrent mode that processes one step at a time with O(1) per-step cost.

The A matrix is initialized using HiPPO-LegS (Legendre polynomials) from Gu et al., 2020, which provides a mathematically principled initialization that captures long-range dependencies. S4D-Lin uses A_n = -1/2 + ni for state dimension n, giving logarithmically-spaced frequencies.

Reference: Gu et al., “On the Parameterization and Initialization of Diagonal State Space Models”, 2022. https://arxiv.org/abs/2206.11893

Constructors

Constructor Summary
S4DLayer(int sequenceLength, int modelDimension, int stateDimension, int expandFactor, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new S4D (Diagonal State Space) layer.

Properties

Property Summary
bool SupportsTraining { get; } Training uses the fused complex diagonal SSM scan’s analytic BPTT node.
int InnerDimension { get; } Gets the inner dimension used for the SSM computation.
int ModelDimension { get; } Gets the model dimension (d_model) of this S4D layer.
int StateDimension { get; } Gets the SSM state dimension (N) controlling the number of independent oscillators.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetAImag() Gets the A parameter (imaginary part) for external inspection.
Tensor<T> GetAReal() Gets the A parameter (real part) for external inspection.
Tensor<T> GetDParameter() Gets the D skip connection parameter for external inspection.
Tensor<T> GetLogDelta() Gets the log-delta (discretization step) parameter for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).