GriffinLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a Griffin language model: embedding + N RGLR blocks with local attention + layer norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → GriffinLanguageModel<T>
For Beginners
Griffin combines fast linear recurrence blocks with local sliding window attention for a hybrid model that processes sequences efficiently while maintaining quality.
How It Works
Griffin from Google DeepMind combines Real-Gated Linear Recurrence (RGLR) blocks with local sliding window attention for a hybrid architecture achieving near-Transformer quality with sub-quadratic complexity. Every Mth block uses local attention instead of RGLR.
Reference: De et al., “Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new GriffinLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
GriffinLanguageModel(NeuralNetworkArchitecture<T> architecture, GriffinOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Griffin blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |