ILocalObjectiveCreditRule<T>

Interfaces · AiDotNet.Interfaces

A credit rule that trains every hidden layer from a local objective — a scalar loss defined on that layer’s own output — instead of a teaching signal routed back from the network output. Gradients never cross a layer boundary below the shared readout: the input to each trainable layer is detached (stop-gradient), so a layer’s parameters are moved only by its own local objective. The trailing ReadoutLayerCount layers form a shared readout that is trained exactly by the network’s output loss.

Implements: ICreditRule<T>

Known implementations: BlockwiseAnchorLocalErrorCreditRule<T>, LayerwiseDistillationCreditRule<T>, SampledTiedReadoutLocalErrorCreditRule<T>, TiedReadoutLocalErrorCreditRule<T>

For Beginners

normal training sends one error signal from the very end of the network all the way back through every layer. Here each layer gets its own small exam (“predict the next token from what you have so far”, or “match the teacher’s layer”) and studies only for that exam. Layers never pass blame to each other, so very deep stacks do not suffer from signals fading or exploding on the way down.

How It Works

This is the form of local credit that applies to sequence models (language models, any network whose layer outputs are [batch, sequence, features]). The classic ICreditRule rules build per-layer feedback matrices over the flattened feature size and a dense one-hot target, which at vocabulary 49,152 and sequence 256 is not representable. A local-objective rule instead states each layer’s loss directly (for example, next-token cross-entropy through the shared readout, or a regression onto a teacher model’s hidden state), and the trainer adds all of them into one objective that is back-propagated once.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; } Parameters owned by the rule itself (for example per-layer projections onto a teacher’s hidden width).
int ReadoutLayerCount { get; } Number of trailing layers of the network that form the shared readout (for a language model: final norm + output head = 2).

Methods

Method Summary
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context) Builds the scalar, tape-recorded local objective for one hidden layer, or returns null to give that layer no local objective (its parameters then receive no gradient this step).
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss) Builds the scalar, tape-recorded output objective on the readout’s prediction.
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients) Called once per step with the gradients of AuxiliaryParameters (missing = zero).