BlockwiseAnchorLocalErrorCreditRule<T>

Models & Types · AiDotNet.NeuralNetworks.CreditAssignment

Blockwise anchor local error: decoupled block-local sequence credit assignment that partitions a deep network into contiguous blocks of at most AnchorStride trainable pre-readout layers, places a shared (full or candidate-sampled) tied-readout next-token loss at the end of every block, and detaches activations (StopGradient) strictly at block boundaries so every layer inside a block receives multi-layer local credit while zero gradient crosses between blocks.

Implements: ILocalObjectiveCreditRule<T>

How It Works

Pure layer-by-layer detachment (LayerStride > 1 on TiedReadoutLocalErrorCreditRule) gives unselected intermediate layers zero gradient, while detaching every single layer (LayerStride = 1) prevents stacked linear-attention / GatedDeltaNet layers from cooperating with a periodic global attention / MLA anchor. Partitioning at anchor boundaries (for example AnchorStride = 4 matching a 3:1 GDN:MLA hybrid block) gives 100% of layers constructive block-local credit with zero end-to-end backpropagation.

When StreamBlocksSequentially is true (the default), LocalObjectiveGradientComputer executes one isolated GradientTape and TensorScope per decoupled block sequentially, bounding peak activation memory to O(AnchorStride) layers with 0% forward recomputation overhead while producing the exact same block-decoupled gradients as single-tape execution.

Constructors

Constructor Summary
BlockwiseAnchorLocalErrorCreditRule(int readoutLayerCount, int anchorStride, int maxCandidates, double localWeight, bool detachReadoutOnHiddenBlocks, bool useDepthProgressiveWeight, bool streamBlocksSequentially, int? seed) Creates a blockwise anchor local-error credit rule.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; }
bool DetachReadoutOnHiddenBlocks { get; } Gets whether intermediate block anchors detach the shared readout’s own parameters.
bool IsExactBackprop { get; }
bool StreamBlocksSequentially { get; } Gets whether decoupled blocks execute sequentially with O(AnchorStride) activation memory.
bool UseDepthProgressiveWeight { get; } Gets whether intermediate block anchor weights scale quadratically with normalized depth.
int AnchorStride { get; } Gets the number of trainable hidden layers per decoupled anchor block.
int MaxCandidates { get; } Gets the candidate vocabulary subset size U (MaxValue for full vocabulary).
int ReadoutLayerCount { get; }
string Name { get; }

Methods

Method Summary
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context)
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss)
bool IsIntermediateAnchor(int hiddenIndex, int hiddenLayerCount) Returns true when hidden trainable layer hiddenIndex (0-based out of hiddenLayerCount) is the anchor terminating an intermediate decoupled block.
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients)
void ComputeTeachingSignals(ICreditAssignmentContext<T> context)
void Initialize(ICreditAssignmentContext<T> context)