Knowledge Distillation

56 public types in this namespace, organized by kind.

Models & Types (35)

Type Summary
AccuracyBasedAdaptiveStrategy<T> Adaptive distillation strategy that adjusts temperature based on student accuracy.
AdaptiveTeacherModel<T> Adaptive teacher model that wraps a base teacher and provides its logits.
AttentionDistillationStrategy<T> Implements attention-based knowledge distillation for transformer models.
CheckpointMetadata Metadata about a saved checkpoint.
ConfidenceBasedAdaptiveStrategy<T> Adaptive distillation strategy that adjusts temperature based on student confidence.
ContrastiveDistillationStrategy<T> Implements Contrastive Representation Distillation (CRD) which transfers knowledge through contrastive learning of sample relationships rather than just matching outputs.
CurriculumTeacherModel<T> Curriculum teacher that wraps a base teacher for curriculum learning scenarios.
DistillationCheckpointManager<T> Manages checkpointing during knowledge distillation training.
DistillationForwardResult<T> Encapsulates the result of a forward pass during knowledge distillation training.
DistillationLossAdapter<T> Presents an IDistillationStrategy as a tape-capable loss so the standard gradient path can train a student with distillation.
DistillationLoss<T> Implements the standard knowledge distillation loss function (Hinton et al., 2015).
DistributedTeacherModel<T> Distributed teacher model that aggregates predictions from multiple distributed workers.
EasyToHardCurriculumStrategy<T> Curriculum distillation strategy that progresses from easy to hard samples.
EnsembleTeacherModel<T> Ensemble teacher model that combines predictions from multiple teacher models.
EntropyBasedAdaptiveStrategy<T> Adaptive distillation strategy that adjusts temperature based on prediction entropy.
FactorTransferDistillationStrategy<T> Factor transfer distillation that transfers knowledge through factorized representations.
FeatureDistillationStrategy<T> Implements feature-based knowledge distillation (FitNets) where the student learns to match the teacher’s intermediate layer representations.
FlowBasedDistillationStrategy<T> Flow-based distillation that matches the information flow between layers.
HardToEasyCurriculumStrategy<T> Curriculum distillation strategy that progresses from hard to easy samples.
HybridDistillationStrategy<T> Hybrid distillation strategy that combines multiple distillation strategies with configurable weights.
IntermediateActivations<T> Stores intermediate layer activations collected during a forward pass.
KnowledgeDistillationTrainer<T> Standard knowledge distillation trainer that uses a fixed teacher model to train a student.
MultiModalTeacherModel<T> Multi-modal teacher that combines multiple input modalities (vision, text, audio).
NeuronSelectivityDistillationStrategy<T> Neuron selectivity distillation that transfers the activation patterns and selectivity of individual neurons.
OnlineTeacherModel<T> Online teacher model that updates its parameters during student training.
PretrainedTeacherModel<T> Pretrained teacher model from external source (e.g., ImageNet, BERT).
ProbabilisticDistillationStrategy<T> Probabilistic distillation that transfers distributional knowledge by matching statistical properties.
QuantizedTeacherModel<T> Quantized teacher model with reduced precision for efficient deployment.
RelationalDistillationStrategy<T> Implements Relational Knowledge Distillation (RKD) which transfers knowledge about relationships between samples rather than individual sample predictions.
SelfDistillationTrainer<T> Implements self-distillation where a model acts as its own teacher to improve calibration and generalization.
SelfTeacherModel<T> Self teacher model that uses the student’s own predictions from earlier training.
SimilarityPreservingStrategy<T> Similarity-preserving distillation that preserves pairwise similarity structure.
TeacherModelWrapper<T> Wraps an existing trained IFullModel to act as a teacher for knowledge distillation.
TransformerTeacherModel<T> Transformer-based teacher model that provides logits from transformer architectures.
VariationalDistillationStrategy<T> Variational distillation based on variational inference principles and information theory.

Base Classes (5)

Type Summary
AdaptiveDistillationStrategyBase<T> Abstract base class for adaptive distillation strategies with performance tracking.
CurriculumDistillationStrategyBase<T> Abstract base class for curriculum distillation strategies with progressive difficulty adjustment.
DistillationStrategyBase<T> Abstract base class for knowledge distillation strategies.
KnowledgeDistillationTrainerBase<T, TInput, TOutput> Abstract base class for all knowledge distillation trainers.
TeacherModelBase<TInput, TOutput, T> Abstract base class for teacher models used in knowledge distillation.

Interfaces (2)

Type Summary
IAdaptiveDistillationStrategy<T> Interface for adaptive distillation strategies that adjust temperature based on student performance.
ICurriculumDistillationStrategy<T> Interface for curriculum distillation strategies that progressively adjust training difficulty.

Enums (11)

Type Summary
AggregationMode Specifies how multiple teacher outputs are combined into a single supervision signal.
AttentionMatchingMode Defines how to match attention patterns between teacher and student.
ContrastiveMode Defines the contrastive learning mode.
CurriculumStrategy Defines the curriculum learning strategy direction.
EnsembleAggregationMode Defines how ensemble predictions are aggregated.
FactorMode
OnlineUpdateMode Defines how an online teacher model is updated during training.
ProbabilisticMode
RelationalDistanceMetric Distance metrics for relational knowledge distillation.
SelectivityMetric
VariationalMode

Options & Configuration (1)

Type Summary
DistillationCheckpointConfig Configuration for distillation checkpoint management.

Helpers & Utilities (2)

Type Summary
DistillationStrategyFactory<T> Factory of named knowledge-distillation strategies.
TeacherModelFactory<T> Factory for creating teacher models from enums and configurations.