EMAVFI<T>
Models & Types · AiDotNet.Video.FrameInterpolation
EMA-VFI: extracting motion and appearance via inter-frame attention for efficient video frame interpolation (Zhang et al., CVPR 2023).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → EMAVFI<T>
For Beginners
EMA-VFI compares small windows of the two frames with attention: what each window attends to in the other frame describes its appearance, and where it attends describes its motion. Both feed a coarse-to-fine motion estimator and a refinement network that produce the in-between frame.
How It Works
The network follows the release (MCG-NJU/EMA-VFI, model/feature_extractor.py, flow_estimation.py, refine.py):
- MotionFormer. Both frames pass three convolution stages, then a cross-scale patch embedding and an
overlapping one lead into two stages of (shifted) window inter-frame attention at 1/8 and 1/16 size. Each window’s queries attend to the other frame’s window: the attended values update its appearance, and the attention-weighted coordinate embedding minus its own is its motion feature.
- Multi-scale flow. A head at 1/16 and one at 1/8 predict the flow pair from time t and a blending mask
from the motion features scaled by t and 1 − t and both frames’ appearance features; the second refines the first’s.
- Refinement. A U-Net over the frames, warped frames, mask, flow and both appearance pyramids warped by
the flows predicts a residual added to the blend, clamped to [0, 1].
Frames are padded by edge replication to multiples of 32 (InputPadder). Any time in [0, 1] can be synthesized.
Training follows train.py and Trainer.py: AdamW (weight decay 1e-4) with a 2000-step linear warm-up from 0 to 2e-4 and cosine decay to 2e-5 over 300 epochs, on the Laplacian-pyramid L1 loss of the refined frame plus half that of each stage’s warped blend.
Constructors
| Constructor | Summary |
|---|---|
EMAVFI(NeuralNetworkArchitecture<T> architecture, EMAVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an EMA-VFI model in native training mode. |
EMAVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, EMAVFIOptions options) |
Creates an EMA-VFI model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |