RecurrentGemmaLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a RecurrentGemma language model: embedding + N RGLR blocks + layer norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → RecurrentGemmaLanguageModel<T>

For Beginners

RecurrentGemma is Google’s production model that uses recurrence instead of attention, giving O(n) complexity and constant memory per token during text generation.

How It Works

RecurrentGemma is Google’s production recurrent language model based on the Griffin architecture. It uses Real-Gated Linear Recurrence (RGLR) blocks for O(n) complexity and O(1) per-token generation.

Reference: Botev et al., “RecurrentGemma: Moving Past Transformers for Efficient Open Language Models”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new RecurrentGemmaLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
RecurrentGemmaLanguageModel(NeuralNetworkArchitecture<T> architecture, RecurrentGemmaOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of RecurrentGemma blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()