IQVFIDistillation<T>
Models & Types · AiDotNet.Video.FrameInterpolation
IQ-VFI’s knowledge-distillation objective (Hu, Jiang, Zhong, Wang and Zheng, CVPR 2024): a pyramid reconstruction loss plus acceleration and motion distillation from a privileged teacher.
For Beginners
A “teacher” model is allowed to peek at the answer frame; the “student” is not. The student is corrected toward the teacher only in the places where it is actually doing worse.
How It Works
Ground-truth intermediate MOTION does not exist, so the flows cannot be supervised directly. IQ-VFI
instead trains a teacher that sees three consecutive frames — including the ground-truth
intermediate frame I_t — and distils its acceleration prior and intermediate flows into a
student that sees only the two input frames.
The mask is the distinguishing idea. Motion distillation is applied only where
||I_t^S - I_t||_1 > ||I_t^T - I_t||_1, i.e. only at pixels the student reconstructs WORSE
than the teacher. Prior schemes distil everywhere, which lets the teacher’s privileged access to
I_t leak in as pressure to imitate it even where the student is already correct — the
overfitting the paper sets out to avoid. Gating on relative error means the teacher is copied only
where it demonstrably knows better.
Constructors
| Constructor | Summary |
|---|---|
IQVFIDistillation() |
Methods
| Method | Summary |
|---|---|
Tensor<T> SelectiveMask(Tensor<T> studentFrame, Tensor<T> teacherFrame, Tensor<T> groundTruth) |
The binary mask M: 1 where the student’s reconstruction error exceeds the teacher’s, 0 elsewhere. |
double AccelerationDistillationLoss(Tensor<T> studentPrior, Tensor<T> teacherPrior) |
Implicit acceleration distillation loss L_IA = ||P^S - P^T||_1. |
double MotionDistillationLoss(Tensor<T> studentForward, Tensor<T> teacherForward, Tensor<T> studentBackward, Tensor<T> teacherBackward, Tensor<T> mask) |
Implicit motion distillation loss L_IM = M * ||f_0t^S - f_0t^T||_1 + M * ||f_1t^S - f_1t^T||_1. |
double PyramidReconstructionLoss(Tensor<T>[] studentLevels, Tensor<T>[] truthLevels) |
Pyramid reconstruction loss L_R = sum_i w_i * ||L_i(student) - L_i(truth)||_1 over image pyramid levels. |
double TotalLoss(double reconstructionLoss, double accelerationLoss, double motionLoss, double reconstructionWeight, double accelerationWeight, double motionWeight) |
The overall objective L_total = w_R * L_R + w_IA * L_IA + w_IM * L_IM. |
Fields
| Field | Summary |
|---|---|
int PaperPyramidLevels |
Number of image pyramid levels in the reconstruction loss. |