RWKV7LanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a full RWKV-7 “Goose” language model: token embedding + N RWKV7Blocks + RMS normalization + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → RWKV7LanguageModel<T>

For Beginners

RWKV-7 is the latest version of the RWKV architecture that combines the best of RNNs and Transformers, achieving linear-time inference with competitive quality to Transformer models.

How It Works

RWKV-7 introduces dynamic state evolution with learnable transition matrices, group normalization on WKV output, and SiLU channel mixing for improved training stability.

Reference: Peng et al., “RWKV-7 Goose with Expressive Dynamic State Evolution”, 2025.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new RWKV7LanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
RWKV7LanguageModel(NeuralNetworkArchitecture<T> architecture, RWKV7Options options, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; }
double FFNMultiplier { get; } Gets the FFN dimension multiplier.
int ModelDimension { get; } Gets the model dimension (d_model).
int NumHeads { get; } Gets the number of attention heads.
int NumLayers { get; } Gets the number of RWKV-7 blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()