FinchLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a full RWKV-6 “Finch” language model: token embedding + N RWKVLayer blocks + RMS normalization + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → FinchLanguageModel<T>
For Beginners
Finch (RWKV-6) builds on Eagle by adding the ability to dynamically decide how much context from previous tokens to blend into the current one. Think of it as a reader who can dynamically adjust their focus: sometimes reading word by word, other times absorbing whole phrases. This adaptive blending helps it better capture complex language patterns while maintaining the same memory-efficient inference as its predecessor.
How It Works
Finch extends Eagle (RWKV-5) with data-dependent token shifting via a LoRA-based mechanism, allowing the model to dynamically adjust how much to blend current and previous tokens.
Reference: Peng et al., “Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence”, 2024.
https://arxiv.org/abs/2404.05892
Example
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new FinchLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
FinchLanguageModel(NeuralNetworkArchitecture<T> architecture, FinchOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Finch blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |