EntropyBasedAdaptiveStrategy<T>

Models & Types · AiDotNet.KnowledgeDistillation.Strategies

Adaptive distillation strategy that adjusts temperature based on prediction entropy.

Inheritance: DistillationStrategyBase<T> → AdaptiveDistillationStrategyBase<T> → EntropyBasedAdaptiveStrategy<T>

For Beginners

Entropy measures how uncertain or “spread out” a probability distribution is. High entropy means the student is uncertain (probabilities are similar across classes). Low entropy means the student is certain (one class has high probability).

How It Works

Entropy Examples:

  • Low Entropy [0.95, 0.03, 0.02]: Student is certain → Class 0 dominates
  • High Entropy [0.35, 0.33, 0.32]: Student is uncertain → All classes similar

Intuition:

  • High Entropy (uncertain) → Student struggling → Lower temp (focus learning)
  • Low Entropy (certain) → Student confident → Higher temp (explore more)

Why Lower Temp for High Entropy? When student is uncertain, we want to provide sharper (lower temp) targets to focus learning on the most important features, rather than soft targets that might reinforce uncertainty.

Best For:

  • Detecting student uncertainty
  • Calibrating overconfident students
  • Datasets where uncertainty patterns are meaningful

Entropy Range:

  • Minimum: 0.0 (completely certain, one class = 1.0)
  • Maximum: 1.0 (normalized, completely uncertain, uniform distribution)
  • Normalized by log(num_classes) to get [0, 1] range

Temperature Mapping: High entropy → high difficulty → lower temperature (sharpen) Low entropy → low difficulty → higher temperature (soften)

Constructors

Constructor Summary
EntropyBasedAdaptiveStrategy(double baseTemperature, double alpha, double minTemperature, double maxTemperature, double adaptationRate) Initializes a new instance of the EntropyBasedAdaptiveStrategy class.

Methods

Method Summary
double ComputeAdaptiveTemperature(Vector<T> studentOutput, Vector<T> teacherOutput) Computes adaptive temperature based on prediction entropy.