S5Layer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Simplified State Space (S5) layer from Smith et al., 2023.
Inheritance: LayerBase<T> → S5Layer<T>
Implements: IShapeContract
For Beginners
S5 is an efficient sequence model that competes with Transformers on long-range tasks while being much more computationally efficient.
Think of S5 as a bank of coupled oscillators processing a signal:
- Each oscillator has a complex frequency (how fast it vibrates) and a decay rate
- Unlike S4D where each feature has its own independent oscillator bank, S5 shares
one oscillator bank across ALL features – this is the “MIMO” (multi-input multi-output) part
- The B matrix controls how each input feature excites the oscillators
- The C matrix controls how oscillator states are combined to produce each output feature
- The D matrix provides a direct skip connection from input to output
The key advantages of S5 over S4D:
- MIMO formulation: Features interact through shared state, enabling richer representations
- Parallel scan: The entire sequence is processed in O(L log L) instead of O(L) sequential steps
- Simpler architecture: Fewer parameters than S4D with comparable or better performance
The “simplified” in S5 refers to replacing S4’s complex NPLR (Normal Plus Low-Rank) structure with a straightforward diagonalization, making the model much easier to implement and reason about while maintaining strong empirical performance.
How It Works
S5 is a multi-input multi-output (MIMO) state space model that uses a diagonalized state matrix with parallel scan for efficient sequence processing. Unlike S4D, which applies independent single-input single-output (SISO) SSMs per feature, S5 uses a single MIMO SSM that couples all input dimensions through shared state dynamics.
The continuous-time MIMO SSM is:
where A is diagonalized via eigendecomposition: A = V * Lambda * V^{-1}, with Lambda containing complex eigenvalues. This diagonalization decouples the state dimensions while retaining the MIMO structure through B and C matrices that mix input/output dimensions with state dimensions.
Discretization uses the Zero-Order Hold (ZOH) method:
The discrete recurrence x_t = A_bar * x_{t-1} + B_bar * u_t is then computed efficiently using a parallel associative scan in O(L log L) time, where L is the sequence length.
The A matrix is initialized using the HiPPO framework (High-order Polynomial Projection Operator) which provides optimal polynomial approximations for continuous signal history. The S5 paper uses the HiPPO-LegS (Legendre-Scaled) initialization, which gives the diagonal eigenvalues Lambda_n = -1/2 + npii, placing them along a vertical line in the left half-plane with logarithmically spaced frequencies. This ensures stable dynamics with a rich frequency spectrum for capturing long-range dependencies.
Reference: Smith et al., “Simplified State Space Layers for Sequence Modeling”, ICLR 2023.
https://arxiv.org/abs/2208.04933
Constructors
| Constructor | Summary |
|---|---|
S5Layer(int sequenceLength, int modelDimension, int stateDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new S5 (Simplified State Space) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (H), the width of input and output at each sequence position. |
int StateDimension { get; } |
Gets the SSM state dimension (N), the number of diagonal complex eigenvalues. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetAImag() |
Gets the A parameter (imaginary part of diagonal eigenvalues) for external inspection. |
Tensor<T> GetAReal() |
Gets the A parameter (real part of diagonal eigenvalues) for external inspection. |
Tensor<T> GetDParameter() |
Gets the D skip connection parameter for external inspection. |
Tensor<T> GetLogDelta() |
Gets the log-delta (discretization step) parameter for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |