ExtendedLSTMLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Extended LSTM (xLSTM) layer from Hochreiter et al., 2024.
Inheritance: LayerBase<T> → ExtendedLSTMLayer<T>
Implements: IShapeContract
For Beginners
xLSTM is a modernized version of the classic LSTM (1997).
The original LSTM was the dominant sequence model for years, but was overtaken by Transformers. xLSTM brings it back by fixing key limitations:
- Exponential gating: Instead of sigmoid (0 to 1), gates use exp() which can amplify
important signals, not just dampen them.
- Matrix memory: Instead of a vector cell, mLSTM uses a matrix. This is like having
a lookup table that maps keys to values, similar to attention but stored as a running sum.
The result: an LSTM that matches Transformer performance at scale while maintaining the efficient O(1) per-step inference of RNNs.
How It Works
xLSTM modernizes the classic LSTM architecture with two key innovations:
- sLSTM (scalar LSTM): Enhanced gating with exponential activation functions and
a new memory mixing mechanism. Uses scalar (diagonal) memory cells.
- mLSTM (matrix LSTM): Replaces the scalar memory cell with a matrix-valued memory,
connecting LSTMs to modern linear attention/state space models.
This layer implements the mLSTM variant, which is the more impactful innovation:
The connection to linear attention: if f_t = 1 and i_t = 1, the matrix cell C_t accumulates k*v outer products exactly like the state matrix in linear attention. The gates allow selective forgetting and input scaling, which is what makes xLSTM competitive.
Reference: Beck et al., “xLSTM: Extended Long Short-Term Memory”, 2024.
https://arxiv.org/abs/2405.04517
Constructors
| Constructor | Summary |
|---|---|
ExtendedLSTMLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Extended LSTM (xLSTM) layer using the mLSTM (matrix memory) variant. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads for the matrix memory. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetForgetGateWeights() |
Gets the forget gate weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |