EMAVFI<T>

Models & Types · AiDotNet.Video.FrameInterpolation

EMA-VFI: extracting motion and appearance via inter-frame attention for efficient video frame interpolation (Zhang et al., CVPR 2023).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → EMAVFI<T>

For Beginners

EMA-VFI compares small windows of the two frames with attention: what each window attends to in the other frame describes its appearance, and where it attends describes its motion. Both feed a coarse-to-fine motion estimator and a refinement network that produce the in-between frame.

How It Works

The network follows the release (MCG-NJU/EMA-VFI, model/feature_extractor.py, flow_estimation.py, refine.py):

  • MotionFormer. Both frames pass three convolution stages, then a cross-scale patch embedding and an

overlapping one lead into two stages of (shifted) window inter-frame attention at 1/8 and 1/16 size. Each window’s queries attend to the other frame’s window: the attended values update its appearance, and the attention-weighted coordinate embedding minus its own is its motion feature.

  • Multi-scale flow. A head at 1/16 and one at 1/8 predict the flow pair from time t and a blending mask

from the motion features scaled by t and 1 − t and both frames’ appearance features; the second refines the first’s.

  • Refinement. A U-Net over the frames, warped frames, mask, flow and both appearance pyramids warped by

the flows predicts a residual added to the blend, clamped to [0, 1].

Frames are padded by edge replication to multiples of 32 (InputPadder). Any time in [0, 1] can be synthesized.

Training follows train.py and Trainer.py: AdamW (weight decay 1e-4) with a 2000-step linear warm-up from 0 to 2e-4 and cosine decay to 2e-5 over 300 epochs, on the Laplacian-pyramid L1 loss of the refined frame plus half that of each stage’s warped blend.

Constructors

Constructor Summary
EMAVFI(NeuralNetworkArchitecture<T> architecture, EMAVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an EMA-VFI model in native training mode.
EMAVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, EMAVFIOptions options) Creates an EMA-VFI model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)