VFIformer<T>
Models & Types · AiDotNet.Video.FrameInterpolation
VFIformer: video frame interpolation with a transformer (Lu, Cui, Tian, Jiang and Jia, CVPR 2022).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → VFIformer<T>
For Beginners
VFIformer first estimates how pixels move between the two frames, warps both frames to the middle, and then uses a transformer, which compares each small window of the image with its surroundings at a coarser scale, to fix the places where the warping went wrong.
How It Works
The network follows the release (dvlab-research/VFIformer: VFIformer_arch.py, transformer_layers.py). IFNet estimates the flows from the midpoint to both frames at half size in three coarse-to-fine blocks; FlowRefineNet_Multis refines them from 1/8 size up with local cosine correlations and warps both frames’ four-level feature pyramids along them. The warped frames are fused and passed to a four-scale transformer encoder-decoder whose windows also attend to a coarser copy of their surroundings (cross-scale window attention); its output is a residual and a blending mask for the two warped frames. The frame is the midpoint. Frames are zero-padded around the centre to multiples of 8·window, as the release’s demo pads.
Training follows the release (models/trainer.py): AdamW at 1e-4 with a weight decay of 1e-4, the rate rising linearly over 2,000 steps and then falling by a cosine to zero over 300 epochs, on the L1 loss plus the ternary census loss. A training input is the pair [6, H, W], optionally followed by the precomputed flows from the midpoint to frame 0 and to frame 1 ([10, H, W], in pixels; the release computes them with LiteFlowNet), which add 0.01 times the mean end-point error of each of IFNet’s three flows. Training crops are multiples of 8·window, and the transformer uses stochastic depth.
Constructors
| Constructor | Summary |
|---|---|
VFIformer(NeuralNetworkArchitecture<T> architecture, VFIformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a VFIformer model in native training mode. |
VFIformer(NeuralNetworkArchitecture<T> architecture, string modelPath, VFIformerOptions options) |
Creates a VFIformer model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |