NeuronSelectivityDistillationStrategy<T>
Models & Types ยท AiDotNet.KnowledgeDistillation.Strategies
Neuron selectivity distillation that transfers the activation patterns and selectivity of individual neurons.
Inheritance: DistillationStrategyBase<T> โ NeuronSelectivityDistillationStrategy<T>
Implements: IIntermediateActivationStrategy<T>
How It Works
For Production Use: This strategy focuses on matching how individual neurons respond to inputs. Some neurons are highly selective (activate strongly for specific patterns), while others are more general. Transferring this selectivity helps the student learn meaningful feature representations.
Key Concept: Neuron selectivity measures how discriminative each neuron is. A highly selective neuron activates strongly for certain inputs and weakly for others. The distribution of selectivity across neurons is important for model performance.
Implementation: We measure selectivity using:
- Activation variance (how much neuron output varies across samples)
- Sparsity (what percentage of time the neuron is active)
- Peak-to-average ratio (how peaked the activation distribution is)
Usage Pattern: This strategy implements both standard output-based distillation and intermediate activation-based selectivity matching. Use as follows:
Standard Usage (via IDistillationStrategy):
With Intermediate Activations (via IIntermediateActivationStrategy):
The selectivityWeight and metric parameters control the intermediate activation loss component.
Constructors
| Constructor | Summary |
|---|---|
NeuronSelectivityDistillationStrategy(string targetLayerName, double selectivityWeight, SelectivityMetric metric, double temperature, double alpha) |
Methods
| Method | Summary |
|---|---|
IntermediateActivations<T> ComputeIntermediateGradient(IntermediateActivations<T> studentIntermediateActivations, IntermediateActivations<T> teacherIntermediateActivations) |
Computes gradients of intermediate activation loss with respect to student activations. |
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the gradient of the base distillation loss on final outputs. |
T ComputeIntermediateLoss(IntermediateActivations<T> studentIntermediateActivations, IntermediateActivations<T> teacherIntermediateActivations) |
Computes intermediate activation loss by matching neuron selectivity between teacher and student. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes the base distillation loss on final outputs. |
T ComputeSelectivityLoss(Vector<T>[] studentActivations, Vector<T>[] teacherActivations) |
Computes neuron selectivity loss by comparing activation patterns across a batch. |