DistillationForwardResult<T>

Models & Types · AiDotNet.KnowledgeDistillation

Encapsulates the result of a forward pass during knowledge distillation training.

For Beginners

When training with knowledge distillation, we need more than just the final output of a model. We also need intermediate layer activations to enable advanced distillation techniques (like feature matching or neuron selectivity). This class packages both the final output and optional intermediate activations together.

How It Works

Components:

  • FinalOutput: The model’s final predictions (e.g., class logits). Shape: [batch_size x num_classes]
  • IntermediateActivations: Internal layer outputs collected during forward pass (optional)

Example:

Used By:

  • Teacher models: Provide reference outputs and activations
  • Student models: Provide outputs to compare against teacher
  • Distillation strategies: Compute loss from both final outputs and intermediate activations

Constructors

Constructor Summary
DistillationForwardResult(Matrix<T> finalOutput) Initializes a new instance of DistillationForwardResult with final output only.
DistillationForwardResult(Matrix<T> finalOutput, IntermediateActivations<T> intermediateActivations) Initializes a new instance of DistillationForwardResult with final output and intermediate activations.

Properties

Property Summary
IntermediateActivations<T> IntermediateActivations { get; } Optional intermediate layer activations collected during the forward pass.
Matrix<T> FinalOutput { get; } The final output of the model’s forward pass.
bool HasIntermediateActivations { get; } Checks if intermediate activations are available.