JambaLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a Jamba language model: embedding + HybridBlockScheduler (Mamba + Attention) + RMS norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → JambaLanguageModel<T>
For Beginners
Jamba combines Mamba’s efficient long-range processing with Transformer attention’s precise token interactions for the best of both worlds.
How It Works
Jamba from AI21 Labs is a hybrid SSM-Attention model that interleaves Mamba blocks with full attention blocks using the Jamba schedule pattern (every Nth block is attention). This achieves strong quality by leveraging attention’s exact retrieval with Mamba’s efficient long-range processing.
Reference: Lieber et al., “Jamba: A Hybrid Transformer-Mamba Language Model”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new JambaLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
JambaLanguageModel(NeuralNetworkArchitecture<T> architecture, JambaOptions options, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Jamba blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |