RelationalDistillationStrategy<T>
Models & Types · AiDotNet.KnowledgeDistillation.Strategies
Implements Relational Knowledge Distillation (RKD) which transfers knowledge about relationships between samples rather than individual sample predictions.
Inheritance: DistillationStrategyBase<T> → RelationalDistillationStrategy<T>
For Beginners
Unlike standard distillation which teaches the student to match the teacher’s predictions for each sample, RKD teaches the student to match the teacher’s understanding of how samples relate to each other.
How It Works
Real-world Analogy: Imagine learning about animals from a zoologist. Standard distillation would teach you to identify each animal individually. Relational distillation teaches you the relationships: “dogs are more similar to wolves than to cats,” “sparrows are closer to robins than to eagles.” This relational knowledge helps you understand the taxonomy and make better predictions about animals you’ve never seen.
Key Insight: The relationships between data points (similarity, ordering, grouping) often contain more information than individual predictions. By preserving these relationships, the student learns a more robust and generalizable representation.
Types of Relations RKD Preserves:
- Distance-wise: Relative distances between samples
- Angle-wise: Angular relationships (geometric structure)
- Ranking: Relative ordering of samples
- Pairwise Similarity: How similar each pair of samples is
Mathematical Foundation: Given samples (x1, x2, x3), standard distillation matches:
- f_student(x1) ≈ f_teacher(x1)
- f_student(x2) ≈ f_teacher(x2)
RKD additionally matches:
- distance(f_student(x1), f_student(x2)) ≈ distance(f_teacher(x1), f_teacher(x2))
- angle(f_student(x1), f_student(x2), f_student(x3)) ≈ angle(f_teacher(x1), f_teacher(x2), f_teacher(x3))
Benefits:
- Better Generalization: Student learns structural knowledge
- Robust to Domain Shift: Relationships more stable than absolute values
- Few-shot Learning: Helps with small training sets
- Metric Learning: Improves learned embeddings
- Complementary: Can combine with standard distillation
When to Use:
- Metric learning tasks (face recognition, image retrieval)
- Few-shot and zero-shot learning
- Domain adaptation (relationships transfer better)
- Learning embeddings/representations
- Classification with fine-grained distinctions
Practical Example: For a batch of 32 images:
- Standard distillation: 32 individual predictions to match
- RKD: 32×32 = 1024 pairwise relationships to match
This richer signal often yields 2-5% accuracy improvement.
Computational Cost:
- Distance-wise: O(n²) for n samples in batch
- Angle-wise: O(n³) for triplets
- Use sampling for large batches (sample k pairs instead of all)
References:
- Park et al. (2019). Relational Knowledge Distillation. CVPR.
- Tung & Mori (2019). Similarity-Preserving Knowledge Distillation. ICCV.
- Peng et al. (2019). Correlation Congruence for Knowledge Distillation. ICCV.
Constructors
| Constructor | Summary |
|---|---|
RelationalDistillationStrategy(double distanceWeight, double angleWeight, double temperature, double alpha, int maxSamplesPerBatch, RelationalDistanceMetric distanceMetric) |
Initializes a new instance of the RelationalDistillationStrategy class. |
Methods
| Method | Summary |
|---|---|
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes gradient of combined output loss and relational loss. |
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) |
Computes combined output loss and relational loss. |
T ComputeRelationalLoss(Vector<T>[] studentEmbeddings, Vector<T>[] teacherEmbeddings) |
Computes relational knowledge distillation loss for a batch of samples. |
void Reset() |
Resets the strategy’s internal state. |