MambaLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a full Mamba language model: token embedding + N MambaBlocks + RMS normalization + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → MambaLanguageModel<T>

For Beginners

Mamba is an efficient alternative to Transformers that processes sequences in linear time instead of quadratic time, making it much faster for long sequences while maintaining competitive quality.

How It Works

This assembles the complete Mamba architecture as described in the original paper. Mamba uses selective state spaces with input-dependent gating to achieve linear-time sequence modeling with competitive quality to Transformers.

Reference: Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new MambaLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
MambaLanguageModel(NeuralNetworkArchitecture<T> architecture, MambaOptions options, ILossFunction<T> lossFunction) Initializes a new instance of the MambaLanguageModel class.

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Mamba blocks.
int StateDimension { get; } Gets the SSM state dimension.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
MambaModelState<T> CreateStepState(int batchSize) Creates fresh decoding state for incremental, O(n) token-by-token generation.
ModelMetadata<T> GetModelMetadata()
Tensor<T> Step(Tensor<T> tokenInput, MambaModelState<T> state) Processes a single token using carried KV-cache state, returning the same logits the full-sequence Predict would produce at that position — but in O(1) work per token rather than reprocessing the prefix.