EagleLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a full RWKV-5 “Eagle” language model: token embedding + N RWKVLayer blocks + RMS normalization + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → EagleLanguageModel<T>
For Beginners
Eagle (RWKV-5) is a language model that processes text like a recurrent network (one token at a time) but can also be trained like a transformer (all tokens in parallel). This gives it transformer-quality text generation with much lower memory usage during inference, since it only needs to store a fixed-size state instead of the entire conversation history.
How It Works
This assembles the complete RWKV-5 “Eagle” architecture. Eagle introduces matrix-valued states with multi-head attention, significantly improving upon RWKV-4’s single-head scalar state design.
Reference: Peng et al., “Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence”, 2024.
https://arxiv.org/abs/2404.05892
Example
var architecture = new NeuralNetworkArchitecture<float>(
InputType.OneDimensional, NeuralNetworkTaskType.TextGeneration,
inputSize: 128, outputSize: 65536);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new EagleLanguageModel<float>(
architecture,
new AiDotNet.NeuralNetworks.Options.EagleOptions
{ VocabSize = 65536, ModelDimension = 2560, NumLayers = 32, NumHeads = 40 }))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
EagleLanguageModel(NeuralNetworkArchitecture<T> architecture, EagleOptions options, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Eagle blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |