LayerwiseDistillationCreditRule<T>
Models & Types · AiDotNet.NeuralNetworks.CreditAssignment
Layerwise distillation as local credit: every hidden layer of the student is trained, with its input detached,
to reproduce one hidden state of a frozen teacher model through a learned linear map
P_k : student width → teacher width; the readout is trained by a blend of the hard-label loss and the
teacher’s soft logits (Hinton et al., 2015). Combines local credit (no gradient crosses a layer boundary below the
readout) with knowledge transfer from an established open-weight model (layer-to-layer hint learning, Romero et al.
2015, FitNets).
Implements: ILocalObjectiveCreditRule<T>
For Beginners
a big trained model (the teacher) shows the small model what each of its layers “looks like” on the same text, and each small-model layer practises imitating one teacher layer on its own. The final layer also learns from the teacher’s guesses about the next word, not just the single correct word.
How It Works
Layer map. The teacher supplies L_t + 1 hidden states per step (index 0 = its embedding output,
index j = output of its block j). Student hidden layer k of H is matched to teacher
state round(k · L_t / (H − 1)), so the student embedding meets the teacher embedding and the last student
hidden layer meets the teacher’s last block, whatever the two depths are.
Target scale. Each teacher state is divided by its per-token RMS before regression, so blocks whose residual
norm grows with depth do not dominate the sum. The local loss is the mean squared error
mean((h_k P_k − t̂_j)²), weighted by hiddenWeight.
Projections.P_k are owned by the rule (they are not model parameters, so they never count toward the
student’s size). They are created on first use (scaled Gaussian, seeded) and trained by plain SGD from their own
gradients each step.
Constructors
| Constructor | Summary |
|---|---|
LayerwiseDistillationCreditRule(Func<Tensor<T>, IReadOnlyList<Tensor<T>>> teacherHidden, Func<Tensor<T>, Tensor<T>> teacherLogits, int readoutLayerCount, double hiddenWeight, double kdAlpha, double temperature, double projectionLearningRate, int? seed, double localTaskWeight) |
Creates the rule. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Tensor<T>> AuxiliaryParameters { get; } |
|
bool IsExactBackprop { get; } |
|
int ReadoutLayerCount { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> BuildLocalObjective(LocalObjectiveContext<T> context) |
|
Tensor<T> BuildOutputObjective(LocalObjectiveStep<T> step, Tensor<T> prediction, Tensor<T> defaultLoss) |
|
void ApplyAuxiliaryGradients(IReadOnlyDictionary<Tensor<T>, Tensor<T>> gradients) |
|
void ComputeTeachingSignals(ICreditAssignmentContext<T> context) |
|
void Initialize(ICreditAssignmentContext<T> context) |