LayerwiseDistillationCreditRule<T>

Models & Types · AiDotNet.NeuralNetworks.CreditAssignment

Layerwise distillation as local credit: every hidden layer of the student is trained, with its input detached, to reproduce one hidden state of a frozen teacher model through a learned linear map P_k : student width → teacher width; the readout is trained by a blend of the hard-label loss and the teacher’s soft logits (Hinton et al., 2015). Combines local credit (no gradient crosses a layer boundary below the readout) with knowledge transfer from an established open-weight model (layer-to-layer hint learning, Romero et al. 2015, FitNets).

Implements: ILocalObjectiveCreditRule<T>

For Beginners

a big trained model (the teacher) shows the small model what each of its layers “looks like” on the same text, and each small-model layer practises imitating one teacher layer on its own. The final layer also learns from the teacher’s guesses about the next word, not just the single correct word.

How It Works

Layer map. The teacher supplies L_t + 1 hidden states per step (index 0 = its embedding output, index j = output of its block j). Student hidden layer k of H is matched to teacher state round(k · L_t / (H − 1)), so the student embedding meets the teacher embedding and the last student hidden layer meets the teacher’s last block, whatever the two depths are.

Target scale. Each teacher state is divided by its per-token RMS before regression, so blocks whose residual norm grows with depth do not dominate the sum. The local loss is the mean squared error mean((h_k P_k − t̂_j)²), weighted by hiddenWeight.

Projections.P_k are owned by the rule (they are not model parameters, so they never count toward the student’s size). They are created on first use (scaled Gaussian, seeded) and trained by plain SGD from their own gradients each step.

Constructors

Constructor Summary
LayerwiseDistillationCreditRule(Func<Tensor<T>, IReadOnlyList<Tensor<T>>> teacherHidden, Func<Tensor<T>, Tensor<T>> teacherLogits, int readoutLayerCount, double hiddenWeight, double kdAlpha, double temperature, double projectionLearningRate, int? seed, double localTaskWeight) Creates the rule.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; }
bool IsExactBackprop { get; }
int ReadoutLayerCount { get; }
string Name { get; }

Methods

Method Summary
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context)
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss)
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients)
void ComputeTeachingSignals(ICreditAssignmentContext<T> context)
void Initialize(ICreditAssignmentContext<T> context)