RWKV7Block<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements a single RWKV-7 “Goose” block with the WKV-7 kernel featuring dynamic state evolution.
Inheritance: LayerBase<T> → RWKV7Block<T>
Implements: IShapeContract
For Beginners
This is one layer in the RWKV-7 model. Think of it as a smart information processor that:
- Reads the current word and blends it with the previous word
- Decides what to remember and what to forget (using learnable transition rules)
- Produces an output that captures both local and long-range context
Unlike Transformers that re-read the entire text each time, RWKV-7 keeps a compact running summary (the “state”) that gets updated with each new word, making it very efficient.
How It Works
RWKV-7 is the seventh generation of the RWKV architecture, introducing expressive dynamic state evolution that replaces the fixed exponential decay of previous versions with learnable, data-dependent transition matrices. This allows the model to dynamically control how information is stored, retained, and forgotten in the recurrent state.
Each block contains two sub-layers with residual connections:
Key innovations over RWKV-6 “Finch”:
- Learnable transition vectors a_t (diagonal state decay) and b_t (additive state injection)
- State evolution: S_t = diag(a_t) * S_{t-1} + b_t * (k_t * v_t), replacing fixed exp decay
- Group normalization on WKV output for stability
- SiLU activation in channel mixing instead of squared ReLU
Reference: Peng et al., “RWKV-7 Goose with Expressive Dynamic State Evolution”, 2025.
Constructors
| Constructor | Summary |
|---|---|
RWKV7Block(int sequenceLength, int modelDimension, int numHeads, double ffnMultiplier, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy, double globalIclrMultiplier) |
Creates a new RWKV-7 block. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Training support is approximate: gradients flow through residual connections and weight gradients are accumulated, but full backpropagation through the WKV-7 recurrent kernel is not yet implemented. |
int FFNDimension { get; } |
Gets the feed-forward network dimension. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers() |
|
Tensor<T> GetPreviousToken() |
Gets the previous token for token-shift continuation. |
Tensor<T> GetRecurrentState() |
Gets the recurrent state for autoregressive continuation. |
void ClearGradients() |
|
void ResetState() |
|
void SetPreviousToken(Tensor<T> token) |
Sets the previous token for token-shift continuation. |
void SetRecurrentState(Tensor<T> state) |
Sets the recurrent state for autoregressive continuation. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |