MixtureOfMambaLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Mixture-of-Mamba layer from Jiang et al., 2025 (arXiv:2501.16295).

Inheritance: LayerBase<T> → MixtureOfMambaLayer<T>

Implements: IShapeContract

For Beginners

Mixture-of-Mamba is like having a team of specialized assistants instead of one generalist.

Imagine you’re processing a long document:

  • You have 8 assistants (experts), each good at different things:
  • Expert 1 might be great at tracking names and entities
  • Expert 2 might excel at following numerical patterns
  • Expert 3 might specialize in temporal/causal relationships
  • etc.
  • For each word, a “router” decides which 2 assistants (top-K=2) should handle it
  • Each chosen assistant processes the word using their own specialized memory (SSM state)
  • The final answer combines the two assistants’ outputs, weighted by confidence

This is more efficient than one giant assistant because:

  • Each expert can specialize in different patterns (divide and conquer)
  • Only 2 out of 8 experts run per token (sparse computation saves resources)
  • Different tokens automatically get routed to the most relevant experts

The “Mamba” part refers to the selective scan mechanism each expert uses, which is a very efficient way to process sequences with learned, input-dependent state transitions.

How It Works

Mixture-of-Mamba combines Mamba’s selective state space model (SSM) with Mixture of Experts (MoE) sparsity. Instead of a single monolithic SSM, the layer maintains multiple “expert” SSM blocks, each with its own A, B, C, D parameters. A learned router selects the top-K experts for each token, allowing different tokens to be processed by different specialized SSM pathways.

The architecture:

The key insight is that different types of sequential patterns benefit from different SSM dynamics. By routing tokens to specialized experts, the model can learn distinct temporal patterns without interference. For example, one expert might specialize in short-range dependencies (fast-decaying A), while another handles long-range dependencies (slow-decaying A). The sparse routing means only top-K experts are active per token, maintaining efficiency.

Reference: Jiang et al., “Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Mixture of Experts”, 2025. https://arxiv.org/abs/2501.16295

Constructors

Constructor Summary
MixtureOfMambaLayer(int sequenceLength, int modelDimension, int numExperts, int topK, int stateDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Mixture-of-Mamba layer.

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension.
int NumExperts { get; } Gets the number of experts.
int StateDimension { get; } Gets the SSM state dimension per expert.
int TopK { get; } Gets the number of active experts per token (top-K).

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetExpertA() Gets the expert A (state transition) parameters for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetRouterWeights() Gets the router weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).