SambaLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a Samba language model: embedding + HybridBlockScheduler (Mamba + sliding window attention) + RMS norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → SambaLanguageModel<T>

For Beginners

Samba alternates between Mamba blocks (for long-range context) and sliding window attention (for precise local interactions) for efficient unlimited-context modeling.

How It Works

Samba from Microsoft Research alternates Mamba blocks with sliding window attention blocks in a regular pattern, combining Mamba’s efficient long-range processing with local attention’s precise token interactions within a fixed window size.

Reference: Ren et al., “Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new SambaLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
SambaLanguageModel(NeuralNetworkArchitecture<T> architecture, SambaOptions options, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Samba blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()