Zamba2LanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a Zamba2 language model: embedding + HybridBlockScheduler (Mamba2 + shared attention with LoRA) + RMS norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → Zamba2LanguageModel<T>

For Beginners

Zamba2 upgrades Zamba with Mamba2 blocks and LoRA-adapted shared attention layers for better efficiency and quality.

How It Works

Zamba2 from Zyphra improves on Zamba by using Mamba2 blocks instead of Mamba1, adding multiple shared attention layers with LoRA adapters for differentiation, and concatenating the original shared attention output with the Mamba block output before each attention invocation.

Reference: Glorioso et al., “The Zamba2 Suite: Technical Report”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    InputType.OneDimensional, NeuralNetworkTaskType.TextGeneration,
    inputSize: 4096, outputSize: 32000);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Zamba2LanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
Zamba2LanguageModel(NeuralNetworkArchitecture<T> architecture, Zamba2Options options, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Zamba2 blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()