MambaLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a full Mamba language model: token embedding + N MambaBlocks + RMS normalization + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → MambaLanguageModel<T>
For Beginners
Mamba is an efficient alternative to Transformers that processes sequences in linear time instead of quadratic time, making it much faster for long sequences while maintaining competitive quality.
How It Works
This assembles the complete Mamba architecture as described in the original paper. Mamba uses selective state spaces with input-dependent gating to achieve linear-time sequence modeling with competitive quality to Transformers.
Reference: Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new MambaLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
MambaLanguageModel(NeuralNetworkArchitecture<T> architecture, MambaOptions options, ILossFunction<T> lossFunction) |
Initializes a new instance of the MambaLanguageModel class. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Mamba blocks. |
int StateDimension { get; } |
Gets the SSM state dimension. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
MambaModelState<T> CreateStepState(int batchSize) |
Creates fresh decoding state for incremental, O(n) token-by-token generation. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Step(Tensor<T> tokenInput, MambaModelState<T> state) |
Processes a single token using carried KV-cache state, returning the same logits the full-sequence Predict would produce at that position — but in O(1) work per token rather than reprocessing the prefix. |