RelationalDistillationStrategy<T>

Models & Types · AiDotNet.KnowledgeDistillation.Strategies

Implements Relational Knowledge Distillation (RKD) which transfers knowledge about relationships between samples rather than individual sample predictions.

Inheritance: DistillationStrategyBase<T> → RelationalDistillationStrategy<T>

For Beginners

Unlike standard distillation which teaches the student to match the teacher’s predictions for each sample, RKD teaches the student to match the teacher’s understanding of how samples relate to each other.

How It Works

Real-world Analogy: Imagine learning about animals from a zoologist. Standard distillation would teach you to identify each animal individually. Relational distillation teaches you the relationships: “dogs are more similar to wolves than to cats,” “sparrows are closer to robins than to eagles.” This relational knowledge helps you understand the taxonomy and make better predictions about animals you’ve never seen.

Key Insight: The relationships between data points (similarity, ordering, grouping) often contain more information than individual predictions. By preserving these relationships, the student learns a more robust and generalizable representation.

Types of Relations RKD Preserves:

  • Distance-wise: Relative distances between samples
  • Angle-wise: Angular relationships (geometric structure)
  • Ranking: Relative ordering of samples
  • Pairwise Similarity: How similar each pair of samples is

Mathematical Foundation: Given samples (x1, x2, x3), standard distillation matches:

  • f_student(x1) ≈ f_teacher(x1)
  • f_student(x2) ≈ f_teacher(x2)

RKD additionally matches:

  • distance(f_student(x1), f_student(x2)) ≈ distance(f_teacher(x1), f_teacher(x2))
  • angle(f_student(x1), f_student(x2), f_student(x3)) ≈ angle(f_teacher(x1), f_teacher(x2), f_teacher(x3))

Benefits:

  • Better Generalization: Student learns structural knowledge
  • Robust to Domain Shift: Relationships more stable than absolute values
  • Few-shot Learning: Helps with small training sets
  • Metric Learning: Improves learned embeddings
  • Complementary: Can combine with standard distillation

When to Use:

  • Metric learning tasks (face recognition, image retrieval)
  • Few-shot and zero-shot learning
  • Domain adaptation (relationships transfer better)
  • Learning embeddings/representations
  • Classification with fine-grained distinctions

Practical Example: For a batch of 32 images:

  • Standard distillation: 32 individual predictions to match
  • RKD: 32×32 = 1024 pairwise relationships to match

This richer signal often yields 2-5% accuracy improvement.

Computational Cost:

  • Distance-wise: O(n²) for n samples in batch
  • Angle-wise: O(n³) for triplets
  • Use sampling for large batches (sample k pairs instead of all)

References:

  • Park et al. (2019). Relational Knowledge Distillation. CVPR.
  • Tung & Mori (2019). Similarity-Preserving Knowledge Distillation. ICCV.
  • Peng et al. (2019). Correlation Congruence for Knowledge Distillation. ICCV.

Constructors

Constructor Summary
RelationalDistillationStrategy(double distanceWeight, double angleWeight, double temperature, double alpha, int maxSamplesPerBatch, RelationalDistanceMetric distanceMetric) Initializes a new instance of the RelationalDistillationStrategy class.

Methods

Method Summary
Matrix<T> ComputeGradient(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes gradient of combined output loss and relational loss.
T ComputeLoss(Matrix<T> studentBatchOutput, Matrix<T> teacherBatchOutput, Matrix<T> trueLabelsBatch) Computes combined output loss and relational loss.
T ComputeRelationalLoss(Vector<T>[] studentEmbeddings, Vector<T>[] teacherEmbeddings) Computes relational knowledge distillation loss for a batch of samples.
void Reset() Resets the strategy’s internal state.