HippoOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for HiPPO (High-order Polynomial Projection Operators) model.
Inheritance: ModelOptions → RegressionOptions<T> → TimeSeriesRegressionOptions<T> → HippoOptions<T>
For Beginners
HiPPO answers a fundamental question in sequence modeling: “How do we optimally remember a continuous history in a fixed-size memory?”
The Key Insight: When processing a sequence, we want our hidden state to be an “optimal summary” of the past. HiPPO shows that by projecting the input history onto polynomial bases (like Legendre polynomials), we can create hidden states that provably capture the history optimally.
How It Works:
- Polynomial Basis: Choose a basis (Legendre, Laguerre, Fourier, etc.)
- Optimal Projection: The state x(t) represents coefficients of the polynomial
approximation of the input history
- Online Update: The state can be updated efficiently as new inputs arrive
- Memory Matrix A: Defines how the state evolves (different for each basis)
The Math (simplified): State Space Model: dx/dt = Ax + Bu
- A is the “HiPPO matrix” derived from the chosen polynomial basis
- x(t) contains coefficients: history ≈ Σ x_i(t) * P_i(τ)
- Different A matrices give different memory properties:
- LegS: Sliding window over recent history
- LegT: Fixed window over [0, t]
- LagT: Exponential decay (older = less weight)
Why HiPPO Matters:
- Provides principled initialization for state space models
- Enables models to handle very long sequences
- Foundation for S4, Mamba, and other modern SSMs
- Mathematically optimal for memory compression
How It Works
HiPPO provides the theoretical foundation for efficient state space models like S4 and Mamba. It defines optimal state matrices for compressing sequential input history into a fixed-size state.
Reference: Gu et al., “HiPPO: Recurrent Memory with Optimal Polynomial Projections”, 2020.
https://arxiv.org/abs/2008.07669
Constructors
| Constructor | Summary |
|---|---|
HippoOptions() |
Initializes a new instance of the HippoOptions class with default values. |
HippoOptions(HippoOptions<T> other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
bool UseGate { get; set; } |
Gets or sets whether the recurrent hidden update uses the paper’s standard sigmoid gate. |
bool UseNormalization { get; set; } |
Gets or sets whether to use normalization between HiPPO layers. |
double DropoutRate { get; set; } |
Gets or sets the dropout rate for regularization. |
double TimeStep { get; set; } |
Gets or sets the LTI discretization step. |
double TimescaleMax { get; set; } |
Gets or sets the maximum timescale for the SSM. |
double TimescaleMin { get; set; } |
Gets or sets the minimum timescale for the SSM. |
int ContextLength { get; set; } |
Gets or sets the context length (input sequence length). |
int ForecastHorizon { get; set; } |
Gets or sets the forecast horizon (prediction length). |
int InitialTime { get; set; } |
Gets or sets the initial time index used by the scale-invariant LegS recurrence. |
int MemorySize { get; set; } |
Gets or sets the number of independent polynomial memories in each recurrent cell. |
int ModelDimension { get; set; } |
Gets or sets the model dimension (d_model). |
int NumLayers { get; set; } |
Gets or sets the number of HiPPO layers. |
int StateDimension { get; set; } |
Gets or sets the state dimension (N in the paper). |
string DiscretizationMethod { get; set; } |
Gets or sets the discretization method for converting continuous to discrete SSM. |
string HippoMethod { get; set; } |
Gets or sets the HiPPO method for state matrix initialization. |