Mamba2Block<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Mamba-2 block using the State Space Duality (SSD) framework from Dao and Gu, 2024.
Inheritance: LayerBase<T> → Mamba2Block<T>
Implements: IShapeContract
For Beginners
Mamba-2 is the faster sequel to Mamba.
Think of it this way:
- Mamba-1 processes each token one at a time (like reading a book word by word)
- Mamba-2 processes chunks of tokens at once (like reading a paragraph at a time)
Within each chunk, it uses a matrix multiplication (like attention) that is very fast on GPUs. Between chunks, it uses the efficient state-passing from Mamba-1. The result: same quality as Mamba-1, but 2-8x faster in practice.
The “multi-head” aspect is similar to multi-head attention in Transformers: each head can focus on different patterns independently.
How It Works
Mamba-2 reveals a deep connection between state space models and structured attention: the selective scan can be expressed as multiplication by a semi-separable matrix, which is equivalent to a form of structured masked attention. This duality enables 2-8x faster computation than Mamba-1 through hardware-efficient block-wise parallel algorithms.
Key differences from Mamba-1:
- Multi-head SSM: state is partitioned into heads (like multi-head attention)
- Block-size chunking: sequence is processed in chunks for parallel within-chunk computation
- Simplified parameterization: A is scalar per head (not per-dimension), B and C are shared across heads
- SSD computation: combines efficient chunked quadratic attention within blocks with linear recurrence across blocks
The architecture per timestep:
Reference: Dao and Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”, 2024.
https://arxiv.org/abs/2405.21060
Constructors
| Constructor | Summary |
|---|---|
Mamba2Block(int sequenceLength, int modelDimension, int stateDimension, int numHeads, int expandFactor, int convKernelSize, int chunkSize, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Mamba-2 block with State Space Duality (SSD) computation. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ChunkSize { get; } |
Gets the chunk size used for block-wise parallel computation. |
int ConvKernelSize { get; } |
Gets the convolution kernel size. |
int HeadDimension { get; } |
Gets the dimension per head. |
int InnerDimension { get; } |
Gets the inner dimension (d_inner = modelDim * expandFactor). |
int ModelDimension { get; } |
Gets the model dimension (d_model) of this Mamba-2 block. |
int NumHeads { get; } |
Gets the number of SSM heads. |
int StateDimension { get; } |
Gets the SSM state dimension (N). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetALogParameter() |
Gets the A_log parameter tensor for external inspection. |
Tensor<T> GetDParameter() |
Gets the D skip connection parameter for external inspection. |
Tensor<T> GetInputProjectionWeights() |
Gets the input projection weights for external inspection or quantization. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection or quantization. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |