SampledTiedReadoutLocalErrorCreditRule<T>

Models & Types · AiDotNet.NeuralNetworks.CreditAssignment

Sampled tied-readout local error: zero-auxiliary-parameter sequence local credit assignment that evaluates each hidden layer’s next-token cross-entropy over a compact candidate subset U ≪ V of the network’s shared readout (all unique batch target tokens plus uniform deterministic negative vocabulary tokens) with readout parameters detached on intermediate layers by default.

Implements: ILocalObjectiveCreditRule<T>

How It Works

Evaluating a full V = 49,152 vocabulary projection on every intermediate hidden layer costs O(L · N · D · V) FLOPs, dominating deep zero-backprop language-model steps. Gathering the U batch-positive + negative columns of the shared readout once per step via IEngine.TensorEmbeddingLookup reduces intermediate readout GEMMs from [N, D] × [D, V] to [N, D] × [D, U] (e.g. 48×–96× smaller at U = 512..1024) while preserving exact target-token coverage and zero auxiliary model parameters.

When MaxCandidates is at least the vocabulary size or the active loss is not class-index CrossEntropyWithLogitsLoss, the rule falls back seamlessly to exact full-readout evaluation (ReadoutLossDetached / ReadoutLoss).

Constructors

Constructor Summary
SampledTiedReadoutLocalErrorCreditRule(int readoutLayerCount, int maxCandidates, double localWeight, bool detachReadoutOnHiddenLayers, int layerStride, bool useDepthProgressiveWeight, int? seed) Creates a sampled tied-readout local credit rule.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; }
bool DetachReadoutOnHiddenLayers { get; } Gets whether intermediate hidden layers detach the shared readout’s own parameters.
bool IsExactBackprop { get; }
bool UseDepthProgressiveWeight { get; } Gets whether local loss weights scale quadratically with normalized layer depth.
int LayerStride { get; } Gets the stride between hidden layers receiving local readout objectives.
int MaxCandidates { get; } Gets the target candidate vocabulary subset size U.
int ReadoutLayerCount { get; }
string Name { get; }

Methods

Method Summary
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context)
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss)
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients)
void ComputeTeachingSignals(ICreditAssignmentContext<T> context)
void Initialize(ICreditAssignmentContext<T> context)