RecurrentGemmaLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a RecurrentGemma language model: embedding + N RGLR blocks + layer norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → RecurrentGemmaLanguageModel<T>
For Beginners
RecurrentGemma is Google’s production model that uses recurrence instead of attention, giving O(n) complexity and constant memory per token during text generation.
How It Works
RecurrentGemma is Google’s production recurrent language model based on the Griffin architecture. It uses Real-Gated Linear Recurrence (RGLR) blocks for O(n) complexity and O(1) per-token generation.
Reference: Botev et al., “RecurrentGemma: Moving Past Transformers for Efficient Open Language Models”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new RecurrentGemmaLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
RecurrentGemmaLanguageModel(NeuralNetworkArchitecture<T> architecture, RecurrentGemmaOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of RecurrentGemma blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |