TiedReadoutLocalErrorCreditRule<T>

Models & Types · AiDotNet.NeuralNetworks.CreditAssignment

Tied-readout local error: a sequence-model form of Local Error Signals (Nøkland & Eidnes, 2019) in which every hidden layer’s local classifier is the network’s own shared readout (for a language model: the final norm and the tied output head) rather than a per-layer learned matrix. Each hidden layer k is trained by λ · loss(readout(h_k), target) with its input detached, so no gradient ever crosses a layer boundary below the readout.

Implements: ILocalObjectiveCreditRule<T>

For Beginners

every layer is handed the model’s own “answer sheet” (the output head) and is graded on how well its output alone predicts the next word. No layer waits for blame to trickle down from the top.

How It Works

Why a shared readout. The classic rule learns a [features, classes] classifier per layer over the flattened layer output, which for a language model is [sequence·width, vocabulary] per layer — not representable at real vocabularies. Re-using the readout costs zero extra parameters, so the model’s parameter count is unchanged, and every layer is asked the same question the network is: predict the next token.

Readout detachment, stride, and depth-progressive weighting. Setting DetachReadoutOnHiddenLayers to true evaluates ReadoutLossDetached on intermediate hidden layers so only the terminal layer trains the shared readout parameters (preventing L − 1 shallow representations from pulling the terminal readout away from final-layer geometry). Setting LayerStride > 1 evaluates local readout supervision every LayerStride blocks anchored below the terminal block, cutting local readout compute linearly.

The last hidden layer feeds the readout directly and is already trained exactly by the output loss; it is given no separate local objective (that would double-count it).

Constructors

Constructor Summary
TiedReadoutLocalErrorCreditRule(int readoutLayerCount, double localWeight, bool detachReadoutOnHiddenLayers, int layerStride, bool useDepthProgressiveWeight) Creates the rule with explicit readout-detachment, layer-stride, and depth-schedule options.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; }
bool DetachReadoutOnHiddenLayers { get; } Gets whether intermediate hidden layers detach the shared readout’s own parameters.
bool IsExactBackprop { get; }
bool UseDepthProgressiveWeight { get; } Gets whether local loss weights scale quadratically with normalized layer depth.
int LayerStride { get; } Gets the stride between hidden layers receiving local readout objectives.
int ReadoutLayerCount { get; }
string Name { get; }

Methods

Method Summary
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context)
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss)
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients)
void ComputeTeachingSignals(ICreditAssignmentContext<T> context)
void Initialize(ICreditAssignmentContext<T> context)