Zamba2LanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a Zamba2 language model: embedding + HybridBlockScheduler (Mamba2 + shared attention with LoRA) + RMS norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → Zamba2LanguageModel<T>
For Beginners
Zamba2 upgrades Zamba with Mamba2 blocks and LoRA-adapted shared attention layers for better efficiency and quality.
How It Works
Zamba2 from Zyphra improves on Zamba by using Mamba2 blocks instead of Mamba1, adding multiple shared attention layers with LoRA adapters for differentiation, and concatenating the original shared attention output with the Mamba block output before each attention invocation.
Reference: Glorioso et al., “The Zamba2 Suite: Technical Report”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(
InputType.OneDimensional, NeuralNetworkTaskType.TextGeneration,
inputSize: 4096, outputSize: 32000);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new Zamba2LanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
Zamba2LanguageModel(NeuralNetworkArchitecture<T> architecture, Zamba2Options options, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Zamba2 blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |