NeuronSelectivityDistillationStrategy<T>

Models & Types ยท AiDotNet.KnowledgeDistillation.Strategies

Neuron selectivity distillation that transfers the activation patterns and selectivity of individual neurons.

Inheritance: DistillationStrategyBase<T> โ†’ NeuronSelectivityDistillationStrategy<T>

Implements: IIntermediateActivationStrategy<T>

How It Works

For Production Use: This strategy focuses on matching how individual neurons respond to inputs. Some neurons are highly selective (activate strongly for specific patterns), while others are more general. Transferring this selectivity helps the student learn meaningful feature representations.

Key Concept: Neuron selectivity measures how discriminative each neuron is. A highly selective neuron activates strongly for certain inputs and weakly for others. The distribution of selectivity across neurons is important for model performance.

Implementation: We measure selectivity using:

  1. Activation variance (how much neuron output varies across samples)
  2. Sparsity (what percentage of time the neuron is active)
  3. Peak-to-average ratio (how peaked the activation distribution is)

Usage Pattern: This strategy implements both standard output-based distillation and intermediate activation-based selectivity matching. Use as follows:

Standard Usage (via IDistillationStrategy):

With Intermediate Activations (via IIntermediateActivationStrategy):

The selectivityWeight and metric parameters control the intermediate activation loss component.

Constructors

Constructor Summary
NeuronSelectivityDistillationStrategy(string targetLayerName, double selectivityWeight, SelectivityMetric metric, double temperature, double alpha)

Methods

Method Summary
IntermediateActivations<T> ComputeIntermediateGradient(IntermediateActivations<T> studentIntermediateActivations, IntermediateActivations<T> teacherIntermediateActivations) Computes gradients of intermediate activation loss with respect to student activations.
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes the gradient of the base distillation loss on final outputs.
T ComputeIntermediateLoss(IntermediateActivations<T> studentIntermediateActivations, IntermediateActivations<T> teacherIntermediateActivations) Computes intermediate activation loss by matching neuron selectivity between teacher and student.
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes the base distillation loss on final outputs.
T ComputeSelectivityLoss(Vector<T>[] studentActivations, Vector<T>[] teacherActivations) Computes neuron selectivity loss by comparing activation patterns across a batch.