DistillationLossAdapter<T>
Models & Types · AiDotNet.KnowledgeDistillation
Presents an IDistillationStrategy as a tape-capable loss so the standard gradient
path can train a student with distillation.
Inheritance: LossFunctionBase<T> → DistillationLossAdapter<T>
How It Works
This is the “tape-aware wrapper around the standard loss” knowledge distillation needs. The distillation loss combines a soft term against the teacher’s logits with a hard term against the true label; that signal cannot travel through the optimizer’s (predicted, actual) loss contract, which carries no teacher. The strategy already computes the loss and its gradient with respect to the student output, but on plain matrices — off the autodiff tape.
How the gradient reaches the parameters. The gradient-computable path backpropagates through
ComputeTapeLoss, so returning the strategy’s gradient from CalculateDerivative
is not enough — the fallback builds a tensor disconnected from the tape, whose parameter gradients
are zero. Instead this uses the standard surrogate-loss construction: with the strategy’s gradient
g treated as a constant, the scalar L = Σ (g · ŷ) is a genuine tape expression over
the prediction ŷ, and ∂L/∂ŷ = g exactly. The tape then carries g back through
the student to real parameter gradients. The scalar’s VALUE is meaningless (it is a surrogate, not
the distillation loss); only its gradient matters, which is the point.
Constructors
| Constructor | Summary |
|---|---|
DistillationLossAdapter(IDistillationStrategy<T> strategy) |
Creates an adapter over a distillation strategy. |
Properties
| Property | Summary |
|---|---|
Vector<T> CurrentTrueLabel { get; set; } |
The true label for the sample currently being trained, or null when labels are unavailable. |
Methods
| Method | Summary |
|---|---|
T CalculateLoss(Vector<T> predicted, Vector<T> actual) |
|
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target) |