Rwkv7Stack<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

A stack of RWKV7Block layers that threads the RWKV-7 value residual between them.

Inheritance: LayerBase<T> → Rwkv7Stack<T>

Implements: IShapeContract

For Beginners

Most layers are a simple chain — each takes the previous one’s output and nothing else. RWKV-7 needs something extra: every layer wants to see what the FIRST layer thought, not just the one below it. A plain chain has nowhere to put that, so instead one object holds the whole run of layers and hands the first layer’s answer along as it goes.

How It Works

RWKV-7 (arXiv:2503.14456) blends every layer’s value projection back toward the FIRST layer’s: v = v + (v_first - v) * sigmoid(v0 + (x_v @ v1) @ v2). That is a cross-layer term, and a layer whose contract is Forward(Tensor) -> Tensor cannot express it — it sees only its own input. The reference implementation solves this by having the PARENT own the loop:

This type is that parent. It owns its blocks and threads v_first as an ordinary local, so the residual is a normal set of tape edges rather than hidden shared state — the gradient flows back through the blend into the producing layer’s value projection like any other data dependency.

This is also the established idiom here rather than a new invention: DenseBlock owns its layers to give each one all previous feature maps, and UNetDiscriminator owns its encoder and decoder to carry skip connections across. Twenty-one composite layers in this codebase follow the same shape.

Constructors

Constructor Summary
Rwkv7Stack(int numLayers, int sequenceLength, int modelDimension, int numHeads, double ffnMultiplier, double globalIclrMultiplier) Creates a stack of RWKV-7 blocks sharing one value-residual chain.

Properties

Property Summary
IReadOnlyList<RWKV7Block<T>> Blocks { get; } The blocks in this stack, in execution order.
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)