MambaBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements a general-purpose Mamba block (Selective State Space Model) from Gu and Dao, 2023.

Inheritance: LayerBase<T> → MambaBlock<T>

Implements: IShapeContract

For Beginners

Mamba is a modern alternative to Transformer attention that is much faster for long sequences.

Think of how attention works in a Transformer:

  • Every token looks at every other token -> O(n^2) cost
  • For 1000 tokens, that’s 1,000,000 comparisons

Mamba works differently:

  • It maintains a “hidden state” that summarizes what it has seen so far
  • Each new token updates this state and produces an output
  • The key innovation is that HOW the state is updated depends on the input
  • This “selective” mechanism lets it remember important tokens and forget irrelevant ones

The result: O(n) linear cost with performance competitive with Transformers. Used by Falcon Mamba 7B, Jamba (AI21), Zamba (Zyphra), and many research models.

How It Works

The Mamba block is the core building block of the Mamba architecture, which processes sequences with O(n) linear time complexity compared to O(n^2) for standard Transformer attention. It uses a selective scan mechanism (S6) where the state space parameters (A, B, C, delta) are input-dependent, allowing the model to selectively propagate or forget information along the sequence dimension.

The block follows the architecture: input projection -> Conv1D -> SiLU -> selective scan -> output gating -> output projection. The selective scan implements the core SSM recurrence:

where A_bar and B_bar are the Zero-Order Hold (ZOH) discretized versions of continuous parameters.

All within-timestep computations use hardware-accelerated Engine tensor operations (SIMD/AVX/GPU), with only the sequential time loop remaining as scalar iteration. This enables full CPU/GPU acceleration for the dominant O(batch * innerDim * stateDim) work per timestep.

Reference: Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2024. https://arxiv.org/abs/2312.00752

Constructors

Constructor Summary
MambaBlock(int sequenceLength, int modelDimension, int stateDimension, int expandFactor, int convKernelSize, int dtRank, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy, double deltaMin, double deltaMax) Creates a new Mamba block.

Properties

Property Summary
bool RequireHiddenStateOutput { get; set; } When true, the forward pass always routes through the decomposed S6Scan path so the per-step hidden states are available via GetHiddenState after the call.
bool SupportsTraining { get; }
int ConvKernelSize { get; } Gets the convolution kernel size used in the depthwise Conv1D.
int DtRank { get; } Gets the rank of the delta (dt) projection, which controls the low-rank bottleneck.
int InnerDimension { get; } Gets the inner dimension (d_inner = modelDim * expandFactor) used for the SSM computation.
int ModelDimension { get; } Gets the model dimension (d_model) of this Mamba block.
int StateDimension { get; } Gets the SSM state dimension (N) controlling the capacity of the hidden state.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetALogParameter() Gets a copy of the A_log parameter tensor (A = -exp(A_log)) for external inspection.
Tensor<T> GetDParameter() Gets a copy of the D skip connection parameter for external inspection.
Tensor<T> GetHiddenState() Gets the current hidden state from the last forward pass, if available.
Tensor<T> GetInputProjectionWeights() Gets a copy of the input projection weights for external inspection or quantization.
Tensor<T> GetOutputProjectionWeights() Gets a copy of the output projection weights for external inspection or quantization.
void ClearGradients()
void ResetState()
void SetDParameter(Tensor<T> values) Overwrites the D (skip-connection) parameter in place.
void SetHiddenState(Tensor<T> state) Sets the initial hidden state for the next forward pass.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).