DistillationStrategyFactory<T>

Helpers & Utilities · AiDotNet.KnowledgeDistillation

Factory of named knowledge-distillation strategies.

How It Works

Each named strategy is a public factory method returning an IDistillationStrategy. The strategy IS the parameter: callers pass a strategy instance directly (e.g. via KnowledgeDistillationOptions.Strategy or ConfigureDistillationStrategy). When no strategy is supplied, response-based distillation (Hinton et al., 2015) is the industry-standard default — see ResolveStrategy.

Methods

Method Summary
IDistillationStrategy<T> CreateAttentionBasedStrategy(Vector<string> attentionLayers, double temperature, double alpha, double attentionWeight) Attention-transfer distillation (Zagoruyko & Komodakis, 2017), for transformer models.
IDistillationStrategy<T> CreateContrastiveStrategy(double temperature, double alpha, ContrastiveMode mode) Contrastive Representation Distillation / CRD (Tian et al., 2020).
IDistillationStrategy<T> CreateFactorTransferStrategy(double temperature, double alpha) Factor Transfer (Kim et al., 2018).
IDistillationStrategy<T> CreateFeatureBasedStrategy(Vector<string> layerPairs, double featureWeight) Feature-based / FitNets distillation (Romero et al., 2014).
IDistillationStrategy<T> CreateFlowBasedStrategy(double temperature, double alpha) Flow of Solution Procedure / FSP (Yim et al., 2017).
IDistillationStrategy<T> CreateHybridStrategy(double temperature, double alpha, Vector<IDistillationStrategy<T>> strategies, Vector<double> strategyWeights) Combined/hybrid distillation.
IDistillationStrategy<T> CreateNeuronSelectivityStrategy(double temperature, double alpha) Neuron Selectivity Transfer / NST (Huang & Wang, 2017).
IDistillationStrategy<T> CreateProbabilisticStrategy(double temperature, double alpha) Probabilistic Knowledge Transfer / PKT (Passalis & Tefas, 2018).
IDistillationStrategy<T> CreateRelationBasedStrategy(double temperature, double alpha) Relational Knowledge Distillation / RKD (Park et al., 2019).
IDistillationStrategy<T> CreateResponseBasedStrategy(double temperature, double alpha) Response-based distillation (Hinton et al., 2015) — the industry-standard default.
IDistillationStrategy<T> CreateSimilarityPreservingStrategy(double temperature, double alpha) Similarity-Preserving Distillation / SP (Tung & Mori, 2019).
IDistillationStrategy<T> CreateVariationalStrategy(double temperature, double alpha) Variational Information Distillation / VID (Ahn et al., 2019).
IDistillationStrategy<T> ResolveStrategy(IDistillationStrategy<T> strategy, double temperature, double alpha) Returns strategy when non-null, otherwise the response-based default (standard Hinton distillation) configured with the supplied temperature and alpha.