SampledTiedReadoutLocalErrorCreditRule<T>
Models & Types · AiDotNet.NeuralNetworks.CreditAssignment
Sampled tied-readout local error: zero-auxiliary-parameter sequence local credit assignment that evaluates
each hidden layer’s next-token cross-entropy over a compact candidate subset U ≪ V of the network’s shared
readout (all unique batch target tokens plus uniform deterministic negative vocabulary tokens) with readout
parameters detached on intermediate layers by default.
Implements: ILocalObjectiveCreditRule<T>
How It Works
Evaluating a full V = 49,152 vocabulary projection on every intermediate hidden layer costs O(L · N · D · V)
FLOPs, dominating deep zero-backprop language-model steps. Gathering the U batch-positive + negative columns
of the shared readout once per step via IEngine.TensorEmbeddingLookup reduces intermediate readout GEMMs from
[N, D] × [D, V] to [N, D] × [D, U] (e.g. 48×–96× smaller at U = 512..1024) while
preserving exact target-token coverage and zero auxiliary model parameters.
When MaxCandidates is at least the vocabulary size or the active loss is not class-index
CrossEntropyWithLogitsLoss, the rule falls back seamlessly to exact
full-readout evaluation (ReadoutLossDetached / ReadoutLoss).
Constructors
| Constructor | Summary |
|---|---|
SampledTiedReadoutLocalErrorCreditRule(int readoutLayerCount, int maxCandidates, double localWeight, bool detachReadoutOnHiddenLayers, int layerStride, bool useDepthProgressiveWeight, int? seed) |
Creates a sampled tied-readout local credit rule. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; } |
|
bool DetachReadoutOnHiddenLayers { get; } |
Gets whether intermediate hidden layers detach the shared readout’s own parameters. |
bool IsExactBackprop { get; } |
|
bool UseDepthProgressiveWeight { get; } |
Gets whether local loss weights scale quadratically with normalized layer depth. |
int LayerStride { get; } |
Gets the stride between hidden layers receiving local readout objectives. |
int MaxCandidates { get; } |
Gets the target candidate vocabulary subset size U. |
int ReadoutLayerCount { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context) |
|
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss) |
|
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients) |
|
void ComputeTeachingSignals(ICreditAssignmentContext<T> context) |
|
void Initialize(ICreditAssignmentContext<T> context) |