Mamba2Block<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Mamba-2 block using the State Space Duality (SSD) framework from Dao and Gu, 2024.

Inheritance: LayerBase<T> → Mamba2Block<T>

Implements: IShapeContract

For Beginners

Mamba-2 is the faster sequel to Mamba.

Think of it this way:

  • Mamba-1 processes each token one at a time (like reading a book word by word)
  • Mamba-2 processes chunks of tokens at once (like reading a paragraph at a time)

Within each chunk, it uses a matrix multiplication (like attention) that is very fast on GPUs. Between chunks, it uses the efficient state-passing from Mamba-1. The result: same quality as Mamba-1, but 2-8x faster in practice.

The “multi-head” aspect is similar to multi-head attention in Transformers: each head can focus on different patterns independently.

How It Works

Mamba-2 reveals a deep connection between state space models and structured attention: the selective scan can be expressed as multiplication by a semi-separable matrix, which is equivalent to a form of structured masked attention. This duality enables 2-8x faster computation than Mamba-1 through hardware-efficient block-wise parallel algorithms.

Key differences from Mamba-1:

  • Multi-head SSM: state is partitioned into heads (like multi-head attention)
  • Block-size chunking: sequence is processed in chunks for parallel within-chunk computation
  • Simplified parameterization: A is scalar per head (not per-dimension), B and C are shared across heads
  • SSD computation: combines efficient chunked quadratic attention within blocks with linear recurrence across blocks

The architecture per timestep:

Reference: Dao and Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”, 2024. https://arxiv.org/abs/2405.21060

Constructors

Constructor Summary
Mamba2Block(int sequenceLength, int modelDimension, int stateDimension, int numHeads, int expandFactor, int convKernelSize, int chunkSize, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Mamba-2 block with State Space Duality (SSD) computation.

Properties

Property Summary
bool SupportsTraining { get; }
int ChunkSize { get; } Gets the chunk size used for block-wise parallel computation.
int ConvKernelSize { get; } Gets the convolution kernel size.
int HeadDimension { get; } Gets the dimension per head.
int InnerDimension { get; } Gets the inner dimension (d_inner = modelDim * expandFactor).
int ModelDimension { get; } Gets the model dimension (d_model) of this Mamba-2 block.
int NumHeads { get; } Gets the number of SSM heads.
int StateDimension { get; } Gets the SSM state dimension (N).

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetALogParameter() Gets the A_log parameter tensor for external inspection.
Tensor<T> GetDParameter() Gets the D skip connection parameter for external inspection.
Tensor<T> GetInputProjectionWeights() Gets the input projection weights for external inspection or quantization.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection or quantization.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).