JambaLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a Jamba language model: embedding + HybridBlockScheduler (Mamba + Attention) + RMS norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → JambaLanguageModel<T>

For Beginners

Jamba combines Mamba’s efficient long-range processing with Transformer attention’s precise token interactions for the best of both worlds.

How It Works

Jamba from AI21 Labs is a hybrid SSM-Attention model that interleaves Mamba blocks with full attention blocks using the Jamba schedule pattern (every Nth block is attention). This achieves strong quality by leveraging attention’s exact retrieval with Mamba’s efficient long-range processing.

Reference: Lieber et al., “Jamba: A Hybrid Transformer-Mamba Language Model”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new JambaLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
JambaLanguageModel(NeuralNetworkArchitecture<T> architecture, JambaOptions options, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Jamba blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()