TiedReadoutLocalErrorCreditRule<T>
Models & Types · AiDotNet.NeuralNetworks.CreditAssignment
Tied-readout local error: a sequence-model form of Local Error Signals (Nøkland & Eidnes, 2019) in which
every hidden layer’s local classifier is the network’s own shared readout (for a language model: the final
norm and the tied output head) rather than a per-layer learned matrix. Each hidden layer k is trained by
λ · loss(readout(h_k), target) with its input detached, so no gradient ever crosses a layer boundary below the
readout.
Implements: ILocalObjectiveCreditRule<T>
For Beginners
every layer is handed the model’s own “answer sheet” (the output head) and is graded on how well its output alone predicts the next word. No layer waits for blame to trickle down from the top.
How It Works
Why a shared readout. The classic rule learns a [features, classes] classifier per layer over the
flattened layer output, which for a language model is [sequence·width, vocabulary] per layer — not
representable at real vocabularies. Re-using the readout costs zero extra parameters, so the model’s parameter
count is unchanged, and every layer is asked the same question the network is: predict the next token.
Readout detachment, stride, and depth-progressive weighting. Setting DetachReadoutOnHiddenLayers
to true evaluates ReadoutLossDetached on intermediate hidden layers so
only the terminal layer trains the shared readout parameters (preventing L − 1 shallow representations from
pulling the terminal readout away from final-layer geometry). Setting LayerStride > 1 evaluates
local readout supervision every LayerStride blocks anchored below the terminal block, cutting local readout
compute linearly.
The last hidden layer feeds the readout directly and is already trained exactly by the output loss; it is given no separate local objective (that would double-count it).
Constructors
| Constructor | Summary |
|---|---|
TiedReadoutLocalErrorCreditRule(int readoutLayerCount, double localWeight, bool detachReadoutOnHiddenLayers, int layerStride, bool useDepthProgressiveWeight) |
Creates the rule with explicit readout-detachment, layer-stride, and depth-schedule options. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; } |
|
bool DetachReadoutOnHiddenLayers { get; } |
Gets whether intermediate hidden layers detach the shared readout’s own parameters. |
bool IsExactBackprop { get; } |
|
bool UseDepthProgressiveWeight { get; } |
Gets whether local loss weights scale quadratically with normalized layer depth. |
int LayerStride { get; } |
Gets the stride between hidden layers receiving local readout objectives. |
int ReadoutLayerCount { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context) |
|
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss) |
|
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients) |
|
void ComputeTeachingSignals(ICreditAssignmentContext<T> context) |
|
void Initialize(ICreditAssignmentContext<T> context) |