IQVFIDistillation<T>

Models & Types · AiDotNet.Video.FrameInterpolation

IQ-VFI’s knowledge-distillation objective (Hu, Jiang, Zhong, Wang and Zheng, CVPR 2024): a pyramid reconstruction loss plus acceleration and motion distillation from a privileged teacher.

For Beginners

A “teacher” model is allowed to peek at the answer frame; the “student” is not. The student is corrected toward the teacher only in the places where it is actually doing worse.

How It Works

Ground-truth intermediate MOTION does not exist, so the flows cannot be supervised directly. IQ-VFI instead trains a teacher that sees three consecutive frames — including the ground-truth intermediate frame I_t — and distils its acceleration prior and intermediate flows into a student that sees only the two input frames.

The mask is the distinguishing idea. Motion distillation is applied only where ||I_t^S - I_t||_1 > ||I_t^T - I_t||_1, i.e. only at pixels the student reconstructs WORSE than the teacher. Prior schemes distil everywhere, which lets the teacher’s privileged access to I_t leak in as pressure to imitate it even where the student is already correct — the overfitting the paper sets out to avoid. Gating on relative error means the teacher is copied only where it demonstrably knows better.

Constructors

Constructor Summary
IQVFIDistillation()

Methods

Method Summary
Tensor<T> SelectiveMask(Tensor<T> studentFrame, Tensor<T> teacherFrame, Tensor<T> groundTruth) The binary mask M: 1 where the student’s reconstruction error exceeds the teacher’s, 0 elsewhere.
double AccelerationDistillationLoss(Tensor<T> studentPrior, Tensor<T> teacherPrior) Implicit acceleration distillation loss L_IA = ||P^S - P^T||_1.
double MotionDistillationLoss(Tensor<T> studentForward, Tensor<T> teacherForward, Tensor<T> studentBackward, Tensor<T> teacherBackward, Tensor<T> mask) Implicit motion distillation loss L_IM = M * ||f_0t^S - f_0t^T||_1 + M * ||f_1t^S - f_1t^T||_1.
double PyramidReconstructionLoss(Tensor<T>[] studentLevels, Tensor<T>[] truthLevels) Pyramid reconstruction loss L_R = sum_i w_i * ||L_i(student) - L_i(truth)||_1 over image pyramid levels.
double TotalLoss(double reconstructionLoss, double accelerationLoss, double motionLoss, double reconstructionWeight, double accelerationWeight, double motionWeight) The overall objective L_total = w_R * L_R + w_IA * L_IA + w_IM * L_IM.

Fields

Field Summary
int PaperPyramidLevels Number of image pyramid levels in the reconstruction loss.