ConsistencyTrainingTrainer<T>

Models & Types · AiDotNet.Diffusion.Distillation

Trainer for consistency training from scratch without a pretrained teacher model.

For Beginners

While consistency distillation needs a “teacher” model to learn from, consistency training learns on its own. It discovers that different amounts of noise added to the same image should all map back to the same clean image. This is like learning to recognize a face from photos taken at different exposure levels — they’re all the same face.

How It Works

Unlike consistency distillation (which requires a pretrained teacher), consistency training learns the consistency function directly from data using a self-consistency loss. The model learns that mapping any point along the same ODE trajectory should produce the same output, without needing teacher guidance. Uses a progressive schedule that increases discretization steps during training.

Reference: Song et al., “Improved Techniques for Training Consistency Models”, ICML 2024 (iCT); Song and Dhariwal, “Consistency Models”, ICML 2023

Constructors

Constructor Summary
ConsistencyTrainingTrainer(double emaDecay, int initialDiscretizationSteps, int finalDiscretizationSteps, double sigmaMin, double sigmaMax) Initializes a new consistency training trainer.

Properties

Property Summary
int CurrentStep { get; } Gets the current training step.
int InitialDiscretizationSteps { get; } Gets the initial discretization step count.

Methods

Method Summary
T ComputeConsistencyLoss(Vector<T> modelOutputAtT, Vector<T> targetOutputAtTPlusOne) Computes the self-consistency loss for a training batch.
T GetSigma(int index, int numSteps) Gets the sigma (noise level) for a given discretization index.
double GetAdaptiveEMADecay(int totalTrainingSteps) Computes the adaptive EMA decay rate based on current discretization.
int GetCurrentDiscretizationSteps(int totalTrainingSteps) Gets the current number of discretization steps based on training progress.
void Step() Advances the training step counter.