S4Options<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for S4 (Structured State Space Sequence Model).
Inheritance: ModelOptions → RegressionOptions<T> → TimeSeriesRegressionOptions<T> → S4Options<T>
For Beginners
S4 is a breakthrough model that showed state space models can match transformers on long-range sequence tasks:
The Key Insight: State space models have linear complexity O(n), but historically couldn’t compete with attention. S4 solves this by:
- Using HiPPO (High-order Polynomial Projection Operators) for state initialization
- Structuring the state matrix A as a diagonal plus low-rank (DPLR) matrix
- Computing convolutions efficiently using FFT
How It Works:
- HiPPO Matrix: Initializes A to optimally compress history into the state
- DPLR Decomposition: A = diagonal + low-rank for efficient computation
- Discretization: Converts continuous SSM to discrete for sequence data
- FFT Convolution: Computes SSM as convolution using O(n log n) FFT
The Math (simplified):
- State update: x’ = Ax + Bu (A is HiPPO-structured)
- Output: y = Cx + Du
- Discretized: x_k = A_bar * x_{k-1} + B_bar * u_k
- For long sequences: compute as convolution K * u using FFT
Advantages:
- Near-linear complexity O(n log n) via FFT
- Excellent on Long Range Arena benchmark
- Handles sequences up to 16K+ tokens
- Foundation for Mamba and other modern SSMs
How It Works
S4 is a foundational state space model that achieves near-linear complexity through structured parameterization of the state transition matrix using the HiPPO framework.
Reference: Gu et al., “Efficiently Modeling Long Sequences with Structured State Spaces”, 2022.
https://arxiv.org/abs/2111.00396
Constructors
| Constructor | Summary |
|---|---|
S4Options() |
Initializes a new instance of the S4Options class with default values. |
S4Options(S4Options<T> other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
bool UseBidirectional { get; set; } |
Gets or sets whether to use bidirectional processing. |
bool UseLowRankCorrection { get; set; } |
Gets or sets whether to use low-rank correction in the DPLR decomposition. |
double DropoutRate { get; set; } |
Gets or sets the dropout rate for regularization. |
double LearningRate { get; set; } |
Gets or sets the peak learning rate for the model’s default Adam optimizer. |
double? HippoMaxLearningRate { get; set; } |
The largest learning rate applied to the HiPPO-derived state-space parameters (B, A and the low-rank P and Q), or null to train them at the model-wide rate. |
int ContextLength { get; set; } |
Gets or sets the context length (input sequence length). |
int ForecastHorizon { get; set; } |
Gets or sets the forecast horizon (prediction length). |
int LowRankRank { get; set; } |
Gets or sets the rank of the low-rank correction. |
int ModelDimension { get; set; } |
Gets or sets the model dimension (d_model). |
int NumLayers { get; set; } |
Gets or sets the number of S4 layers. |
int StateDimension { get; set; } |
Gets or sets the state dimension (N in the paper). |
int WarmupSteps { get; set; } |
Gets or sets the number of optimizer steps over which the learning rate ramps linearly from zero to LearningRate. |
string DiscretizationMethod { get; set; } |
Gets or sets the discretization method for converting continuous to discrete SSM. |
string HippoMethod { get; set; } |
Gets or sets the HiPPO method for state matrix initialization. |