XVFI<T>

Models & Types · AiDotNet.Video.FrameInterpolation

XVFI: extreme video frame interpolation (Sim, Oh and Kim, ICCV 2021).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → XVFI<T>

For Beginners

XVFI handles very large motion in 4K video by estimating motion on a pyramid of ever smaller copies of the frames, starting at the coarsest, and refining it one level at a time. The same small network is reused at every level, so you can add levels at test time for bigger motion.

How It Works

The network follows the release (JihyongOh/XVFI, XVFInet.py). Frames are mapped to [−1, 1]. A shared feature extractor reduces both frames by the scale factor, and one shared stride-2 convolution builds the feature at every scale level. The same VFInet runs from the lowest level up: BiOF-I estimates the bidirectional flows (from the features at the lowest level, else refining the level below’s 2x-upsampled flows with warped features), and below level 0 only the flows are passed on. At level 0 (and at every level in training) complementary flow reversal turns them into flows from time t by z-weighted Gaussian forward warping, a refinement stack corrects those, and BiOF-T blends the warped frames by a refinement U-Net’s occlusion map and adds its residual. Sides are zero-padded on the right and bottom to multiples of 2^S·scale·4, as the release’s test pads. Any time in [0, 1] can be synthesized, and the test depth can exceed the training depth.

Training follows main.py: Adam (betas 0.9 and 0.999, no weight decay) at 1e-4, multiplied by 0.25 at epochs 100, 150 and 180, on the L1 reconstruction loss averaged over every level’s frame against the bicubically downscaled ground truth, plus 0.5 times the edge-aware smoothness of level 0’s refined flows. Training crops are multiples of 2^S·scale·4.

Constructors

Constructor Summary
XVFI(NeuralNetworkArchitecture<T> architecture, XVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an XVFI model in native training mode.
XVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, XVFIOptions options) Creates an XVFI model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)
void EndEpoch() Ends a training epoch, the release’s scheduler.step(): MultiStepLR multiplies the rate by LearningRateDecay at each of LearningRateMilestones.