RWKV7LanguageModelOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the RWKV-7 “Goose” language model.
Inheritance: ModelOptions → RWKV7LanguageModelOptions<T>
For Beginners
RWKV-7 is a text generation model that processes text in linear time:
Key Advantages:
- Linear time complexity: O(n) vs O(n^2) for Transformers
- Constant memory per token during generation
- Competitive quality with Transformer models of similar size
Architecture:
- Token embedding converts words to vectors
- N RWKV-7 blocks process the sequence, each with:
- Time mixing: WKV-7 kernel with dynamic state evolution
- Channel mixing: SiLU-gated feed-forward network
- RMS normalization for stability
- LM head projects to vocabulary logits
Key Innovation (WKV-7): Instead of fixed exponential decay (RWKV-6), the state transition matrices a_t and b_t are learnable and data-dependent: S_t = diag(sigmoid(a_t)) * S_{t-1} + sigmoid(b_t) * outer(k_t, v_t)
This allows the model to dynamically decide what to remember and forget.
Typical Model Sizes:
- 0.1B: modelDim=768, numLayers=12, numHeads=12
- 1.5B: modelDim=2048, numLayers=24, numHeads=32
- 7B: modelDim=4096, numLayers=32, numHeads=64
How It Works
RWKV-7 is the seventh generation of the RWKV architecture, introducing expressive dynamic state evolution that replaces the fixed exponential decay of previous versions with learnable, data-dependent transition matrices.
Reference: Peng et al., “RWKV-7 Goose with Expressive Dynamic State Evolution”, 2025.
Constructors
| Constructor | Summary |
|---|---|
RWKV7LanguageModelOptions() |
Initializes a new instance with default values. |
RWKV7LanguageModelOptions(RWKV7LanguageModelOptions<T> other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
double DropoutRate { get; set; } |
Gets or sets the dropout rate for regularization. |
double FFNMultiplier { get; set; } |
Gets or sets the FFN expansion multiplier. |
int MaxSequenceLength { get; set; } |
Gets or sets the maximum sequence length. |
int ModelDimension { get; set; } |
Gets or sets the model dimension (d_model). |
int NumHeads { get; set; } |
Gets or sets the number of heads per block. |
int NumLayers { get; set; } |
Gets or sets the number of RWKV-7 blocks. |
int VocabSize { get; set; } |
Gets or sets the vocabulary size. |