RWKV7Block<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements a single RWKV-7 “Goose” block with the WKV-7 kernel featuring dynamic state evolution.

Inheritance: LayerBase<T> → RWKV7Block<T>

Implements: IShapeContract

For Beginners

This is one layer in the RWKV-7 model. Think of it as a smart information processor that:

  1. Reads the current word and blends it with the previous word
  2. Decides what to remember and what to forget (using learnable transition rules)
  3. Produces an output that captures both local and long-range context

Unlike Transformers that re-read the entire text each time, RWKV-7 keeps a compact running summary (the “state”) that gets updated with each new word, making it very efficient.

How It Works

RWKV-7 is the seventh generation of the RWKV architecture, introducing expressive dynamic state evolution that replaces the fixed exponential decay of previous versions with learnable, data-dependent transition matrices. This allows the model to dynamically control how information is stored, retained, and forgotten in the recurrent state.

Each block contains two sub-layers with residual connections:

Key innovations over RWKV-6 “Finch”:

  • Learnable transition vectors a_t (diagonal state decay) and b_t (additive state injection)
  • State evolution: S_t = diag(a_t) * S_{t-1} + b_t * (k_t * v_t), replacing fixed exp decay
  • Group normalization on WKV output for stability
  • SiLU activation in channel mixing instead of squared ReLU

Reference: Peng et al., “RWKV-7 Goose with Expressive Dynamic State Evolution”, 2025.

Constructors

Constructor Summary
RWKV7Block(int sequenceLength, int modelDimension, int numHeads, double ffnMultiplier, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy, double globalIclrMultiplier) Creates a new RWKV-7 block.

Properties

Property Summary
bool SupportsTraining { get; } Training support is approximate: gradients flow through residual connections and weight gradients are accumulated, but full backpropagation through the WKV-7 recurrent kernel is not yet implemented.
int FFNDimension { get; } Gets the feed-forward network dimension.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers()
Tensor<T> GetPreviousToken() Gets the previous token for token-shift continuation.
Tensor<T> GetRecurrentState() Gets the recurrent state for autoregressive continuation.
void ClearGradients()
void ResetState()
void SetPreviousToken(Tensor<T> token) Sets the previous token for token-shift continuation.
void SetRecurrentState(Tensor<T> state) Sets the recurrent state for autoregressive continuation.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).