XVFI<T>
Models & Types · AiDotNet.Video.FrameInterpolation
XVFI: extreme video frame interpolation (Sim, Oh and Kim, ICCV 2021).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → XVFI<T>
For Beginners
XVFI handles very large motion in 4K video by estimating motion on a pyramid of ever smaller copies of the frames, starting at the coarsest, and refining it one level at a time. The same small network is reused at every level, so you can add levels at test time for bigger motion.
How It Works
The network follows the release (JihyongOh/XVFI, XVFInet.py). Frames are mapped to [−1, 1]. A shared feature extractor reduces both frames by the scale factor, and one shared stride-2 convolution builds the feature at every scale level. The same VFInet runs from the lowest level up: BiOF-I estimates the bidirectional flows (from the features at the lowest level, else refining the level below’s 2x-upsampled flows with warped features), and below level 0 only the flows are passed on. At level 0 (and at every level in training) complementary flow reversal turns them into flows from time t by z-weighted Gaussian forward warping, a refinement stack corrects those, and BiOF-T blends the warped frames by a refinement U-Net’s occlusion map and adds its residual. Sides are zero-padded on the right and bottom to multiples of 2^S·scale·4, as the release’s test pads. Any time in [0, 1] can be synthesized, and the test depth can exceed the training depth.
Training follows main.py: Adam (betas 0.9 and 0.999, no weight decay) at 1e-4, multiplied by 0.25 at epochs 100, 150 and 180, on the L1 reconstruction loss averaged over every level’s frame against the bicubically downscaled ground truth, plus 0.5 times the edge-aware smoothness of level 0’s refined flows. Training crops are multiples of 2^S·scale·4.
Constructors
| Constructor | Summary |
|---|---|
XVFI(NeuralNetworkArchitecture<T> architecture, XVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an XVFI model in native training mode. |
XVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, XVFIOptions options) |
Creates an XVFI model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |
|
void EndEpoch() |
Ends a training epoch, the release’s scheduler.step(): MultiStepLR multiplies the rate by LearningRateDecay at each of LearningRateMilestones. |