Rwkv7Stack<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
A stack of RWKV7Block layers that threads the RWKV-7 value residual between them.
Inheritance: LayerBase<T> → Rwkv7Stack<T>
Implements: IShapeContract
For Beginners
Most layers are a simple chain — each takes the previous one’s output and nothing else. RWKV-7 needs something extra: every layer wants to see what the FIRST layer thought, not just the one below it. A plain chain has nowhere to put that, so instead one object holds the whole run of layers and hands the first layer’s answer along as it goes.
How It Works
RWKV-7 (arXiv:2503.14456) blends every layer’s value projection back toward the FIRST layer’s:
v = v + (v_first - v) * sigmoid(v0 + (x_v @ v1) @ v2). That is a cross-layer term, and a
layer whose contract is Forward(Tensor) -> Tensor cannot express it — it sees only its
own input. The reference implementation solves this by having the PARENT own the loop:
This type is that parent. It owns its blocks and threads v_first as an ordinary local, so the residual is a normal set of tape edges rather than hidden shared state — the gradient flows back through the blend into the producing layer’s value projection like any other data dependency.
This is also the established idiom here rather than a new invention: DenseBlock owns its
layers to give each one all previous feature maps, and UNetDiscriminator owns its encoder
and decoder to carry skip connections across. Twenty-one composite layers in this codebase follow
the same shape.
Constructors
| Constructor | Summary |
|---|---|
Rwkv7Stack(int numLayers, int sequenceLength, int modelDimension, int numHeads, double ffnMultiplier, double globalIclrMultiplier) |
Creates a stack of RWKV-7 blocks sharing one value-residual chain. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<RWKV7Block<T>> Blocks { get; } |
The blocks in this stack, in execution order. |
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |