ExtendedLSTMLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Extended LSTM (xLSTM) layer from Hochreiter et al., 2024.

Inheritance: LayerBase<T> → ExtendedLSTMLayer<T>

Implements: IShapeContract

For Beginners

xLSTM is a modernized version of the classic LSTM (1997).

The original LSTM was the dominant sequence model for years, but was overtaken by Transformers. xLSTM brings it back by fixing key limitations:

  1. Exponential gating: Instead of sigmoid (0 to 1), gates use exp() which can amplify

important signals, not just dampen them.

  1. Matrix memory: Instead of a vector cell, mLSTM uses a matrix. This is like having

a lookup table that maps keys to values, similar to attention but stored as a running sum.

The result: an LSTM that matches Transformer performance at scale while maintaining the efficient O(1) per-step inference of RNNs.

How It Works

xLSTM modernizes the classic LSTM architecture with two key innovations:

  1. sLSTM (scalar LSTM): Enhanced gating with exponential activation functions and

a new memory mixing mechanism. Uses scalar (diagonal) memory cells.

  1. mLSTM (matrix LSTM): Replaces the scalar memory cell with a matrix-valued memory,

connecting LSTMs to modern linear attention/state space models.

This layer implements the mLSTM variant, which is the more impactful innovation:

The connection to linear attention: if f_t = 1 and i_t = 1, the matrix cell C_t accumulates k*v outer products exactly like the state matrix in linear attention. The gates allow selective forgetting and input scaling, which is what makes xLSTM competitive.

Reference: Beck et al., “xLSTM: Extended Long Short-Term Memory”, 2024. https://arxiv.org/abs/2405.04517

Constructors

Constructor Summary
ExtendedLSTMLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Extended LSTM (xLSTM) layer using the mLSTM (matrix memory) variant.

Properties

Property Summary
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of heads for the matrix memory.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetForgetGateWeights() Gets the forget gate weights for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).