S4<T>

Models & Types · AiDotNet.Finance.Forecasting.StateSpace

S4 (Structured State Space Sequence Model) for time series forecasting.

Inheritance: NeuralNetworkBase<T> → FinancialModelBase<T> → ForecastingModelBase<T> → S4<T>

For Beginners

S4 is a breakthrough model that showed state space models can match transformers on long-range sequence tasks:

The Core Idea: S4 uses a “structured” state space model where:

  1. The A matrix is initialized using HiPPO (optimal for compressing history)
  2. A is decomposed as diagonal + low-rank for efficiency (DPLR)
  3. The computation is done as a convolution using FFT (O(n log n))

Why S4 Works:

  • HiPPO initialization: The state optimally compresses the input history
  • DPLR structure: Allows efficient computation while keeping HiPPO’s benefits
  • FFT convolution: Converts sequential computation to parallel

The Math (simplified): State Space Model: x’(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t)

  • A: State transition (N x N HiPPO matrix, DPLR decomposed)
  • B: Input projection (N x 1)
  • C: Output projection (1 x N)
  • D: Direct feedthrough (skip connection)

Discretized for sequences: x_k = A_bar * x_{k-1} + B_bar * u_k Computed as convolution: y = K * u where K is the SSM kernel

Key Benefits:

  • O(n log n) complexity via FFT (vs O(n^2) for attention)
  • Handles very long sequences (16K+ tokens)
  • Foundation for modern SSMs (Mamba, H3, etc.)
  • Strong performance on Long Range Arena benchmark

How It Works

S4 is a foundational state space model that achieves near-linear complexity through structured parameterization of the state transition matrix using the HiPPO framework.

Reference: Gu et al., “Efficiently Modeling Long Sequences with Structured State Spaces”, 2022. https://arxiv.org/abs/2111.00396

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 256, inputWidth: 1, inputDepth: 1, outputSize: 24);
var model = new S4<double>(architecture);
var onnxModel = new S4<double>(architecture, "s4.onnx");

Constructors

Constructor Summary
S4(NeuralNetworkArchitecture<T> architecture, S4Options<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance of the S4 model in native mode for training.
S4(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, S4Options<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance of the S4 model in ONNX mode for inference.

Properties

Property Summary
bool IsChannelIndependent { get; }
bool SupportsTraining { get; } Gets whether the model supports training (native mode only).
bool UseLowRankCorrection { get; } Gets whether low-rank correction is used in DPLR decomposition.
bool UseNativeMode { get; }
int ContextLength { get; } Gets the input context length for the model.
int ForecastHorizon { get; } Gets the forecast horizon (number of future steps to predict).
int NumFeatures { get; }
int PatchSize { get; }
int PredictionHorizon { get; }
int SequenceLength { get; }
int StateDimension { get; } Gets the state dimension of the SSM.
int Stride { get; }
string HippoMethod { get; } Gets the HiPPO method used for state initialization.

Methods

Method Summary
Dictionary<string, T> Evaluate(Tensor<T> predictions, Tensor<T> actuals) Evaluates forecast quality against actual values.
Dictionary<string, T> GetFinancialMetrics() Gets financial-specific metrics about the model.
ModelMetadata<T> GetModelMetadata() Gets metadata about the S4 model.
Tensor<T> ApplyInstanceNormalization(Tensor<T> input) Applies instance normalization to the input.
Tensor<T> AutoregressiveForecast(Tensor<T> input, int steps) Performs autoregressive forecasting step by step.
Tensor<T> Forward(Tensor<T> input) Performs the forward pass through all layers.
ValueTuple<Tensor<T>, Tensor<T>, Tensor<T>> ForecastWithIntervals(Tensor<T> input, double confidenceLevel) Generates forecasts with prediction intervals for uncertainty quantification.