RWKV4LanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a full RWKV-4 language model: token embedding + N RWKVLayer blocks + layer normalization + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → RWKV4LanguageModel<T>
For Beginners
RWKV-4 is a language model that generates text like GPT, but runs much faster because it processes text in linear time instead of quadratic time.
How it works:
- Each word is converted to a vector (embedding)
- Multiple RWKV layers process the vectors, building understanding of context
- The output is probabilities for what the next word should be
What makes it special:
- Processes text in linear time (twice as long text takes twice as long, not four times)
- Uses constant memory per token during generation
- First RWKV version to achieve competitive quality with Transformers
Real-world examples: RWKV-4 models from 169M to 14B parameters.
How It Works
This assembles the complete RWKV-4 architecture as described in the original paper:
RWKV-4 is the first widely-adopted version of the RWKV architecture. It replaces standard multi-head attention with a linear-complexity WKV (Weighted Key Value) mechanism that uses fixed exponential decay to weight past tokens. The channel mixing sub-layer replaces the standard FFN with a squared ReLU gating mechanism.
Key characteristics of RWKV-4:
- Fixed learned time decay per channel (not data-dependent)
- Single-head WKV attention (no matrix-valued states)
- Squared ReLU channel mixing: sigmoid(r) * (W_v * max(k, 0)^2)
- Token shift mixing with fixed learned coefficients
- O(n) time complexity and O(1) memory per token during generation
Reference: Peng et al., “RWKV: Reinventing RNNs for the Transformer Era”, 2023.
https://arxiv.org/abs/2305.13048
Constructors
| Constructor | Summary |
|---|---|
RWKV4LanguageModel(NeuralNetworkArchitecture<T> architecture, RWKV4Options options, ILossFunction<T> lossFunction) |
Creates an RWKV-4 language model using native library layers. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of RWKV-4 blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |