Mamba2LanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a Mamba-2 language model: token embedding + N Mamba2Blocks + layer normalization + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → Mamba2LanguageModel<T>

For Beginners

Mamba-2 is a faster version of the Mamba language model that discovers a mathematical connection between state-space models and transformers. This insight allows it to use optimized matrix multiplication hardware (like tensor cores on GPUs) for a 2-8x speedup over the original Mamba, while maintaining the same constant memory advantage during text generation.

How It Works

Mamba-2 improves upon the original Mamba architecture by replacing the selective scan with a structured state space duality (SSD) formulation that enables more efficient hardware utilization.

Reference: Dao and Gu, “Transformers are SSMs”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Mamba2LanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
Mamba2LanguageModel(NeuralNetworkArchitecture<T> architecture, Mamba2Options options, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Mamba-2 blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()