ZambaLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a Zamba language model: embedding + HybridBlockScheduler (Mamba + shared attention) + RMS norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → ZambaLanguageModel<T>
For Beginners
Zamba uses mostly Mamba blocks with a single shared attention layer reused at regular intervals, achieving strong quality with fewer parameters.
How It Works
Zamba from Zyphra uses a hybrid architecture where Mamba blocks are the backbone and a single shared attention layer is interleaved at regular intervals. The shared attention weights reduce parameter count while retaining attention’s retrieval capabilities.
Reference: Glorioso et al., “Zamba: A Compact 7B SSM Hybrid Model”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(
InputType.OneDimensional, NeuralNetworkTaskType.TextGeneration,
inputSize: 4096, outputSize: 32000);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new ZambaLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
ZambaLanguageModel(NeuralNetworkArchitecture<T> architecture, ZambaOptions options, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Zamba blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |