VFIT<T>

Models & Types · AiDotNet.Video.FrameInterpolation

VFIT: the video frame interpolation transformer (Shi, Xu, Liu, Chen and Yang, CVPR 2022).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → VFIT<T>

For Beginners

VFIT looks at two frames before and two after the missing one. A transformer compares small windows within each frame and the same pixel across the four frames, and the result decides, for every output pixel, which nearby input pixels to blend and by how much.

How It Works

The network follows the release (zhshi0816/Video-Frame-Interpolation-Transformer: model/VFIT_B.py, Sep_STS_Encoder.py, Sep_STS_Layer.py, cupy_module/adacof.py). The four frames are stacked into a clip and each sample’s per-channel mean subtracted; a 3x3x3 stem and residual block at full size feed four stages that each halve the size with a strided 3D convolution and run Sep-STS blocks: spatial attention in 8x8 windows (shifted by half in every second block) and then temporal attention across each pixel’s frames, followed by an MLP. A 3D U-Net decoder of transposed convolutions, joined to the encoder by concatenation, gives features at 1/8, 1/4 and 1/2 size, from which AdaCoF synthesis predicts, per input frame, 5x5 kernels of weights and vertical and horizontal offsets at 1/4, 1/2 and full size, plus occlusion maps softmaxed over the frames; each frame, resized, is warped by its kernels and the gated sum is added to the coarser prediction upsampled. The frame is the midpoint; frames need no padding.

The input is the window of NumInputFrames frames stacked along the channels, earliest first ([3·frames, H, W] or batched). A frame pair alone is accepted as the window with each frame repeated over its half, which is what Interpolate passes.

Training follows main.py: Adamax (betas 0.9 and 0.999) on the L1 loss of the full-size frame, the rate stepped through 2e-4, 1e-4, 5e-5, 2.5e-5, 5e-6 and 1e-6 at epochs 40, 60, 75, 85 and 95 of 100, batch 4 on 256x256 crops.

Constructors

Constructor Summary
VFIT(NeuralNetworkArchitecture<T> architecture, VFITOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a VFIT model in native training mode.
VFIT(NeuralNetworkArchitecture<T> architecture, string modelPath, VFITOptions options) Creates a VFIT model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)