RWKVLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the RWKV (Receptance Weighted Key Value) layer, a linear attention RNN from Peng et al., 2024.

Inheritance: LayerBase<T> → RWKVLayer<T>

Implements: IShapeContract

For Beginners

RWKV is like a clever hybrid between a Transformer and an RNN.

Imagine you’re summarizing a conversation:

  • A Transformer re-reads the entire conversation for every new sentence (expensive but thorough)
  • An RNN keeps a running summary and just adds new info (cheap but may forget)
  • RWKV keeps a running summary (like an RNN) but uses a smart weighting scheme

so recent information is weighted more heavily, like how you naturally pay more attention to what was just said

The “token shift” mechanism is like looking at both the current word and the previous word to decide what’s important - a simple but effective trick.

Used by RWKV Foundation models (Eagle v5, Finch v6, Goose v7) which achieve competitive performance with Transformers at much lower inference cost.

How It Works

RWKV combines the training parallelism of Transformers with the efficient inference of RNNs. It uses a linear attention mechanism with data-dependent decay, avoiding the quadratic complexity of standard attention while maintaining competitive quality for language modeling.

The architecture consists of two mixing modules per layer:

RWKV v6 (Finch) adds data-dependent linear interpolation for the token-shift mixing coefficients, making mu_r, mu_k, mu_v input-dependent rather than fixed learned parameters.

Reference: Peng et al., “Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence”, 2024. https://arxiv.org/abs/2404.05892

Constructors

Constructor Summary
RWKVLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new RWKV layer.

Properties

Property Summary
bool SupportsTraining { get; } Training IS supported.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetOutputWeights() Gets a copy of the output projection weights for external inspection.
Tensor<T> GetReceptanceWeights() Gets a copy of the receptance projection weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).