BiMVFI<T>
Models & Types · AiDotNet.Video.FrameInterpolation
BiM-VFI: bidirectional motion field-guided frame interpolation for video with non-uniform motions (Seo, Oh and Kim, CVPR 2025).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → BiMVFI<T>
For Beginners
BiM-VFI describes how each pixel moves between the frames by a speed ratio and an angle, estimates the motion from the in-between moment back to each frame from coarse to fine, and blends the two warped frames into the in-between frame.
How It Works
The network follows the release (KAIST-VICLab/BiM-VFI, modules/components/bim_vfi):
- Bidirectional motion field (BiM). The motion from the intermediate time to the two frames is described
by the ratio r of their magnitudes and the angle φ between them. At inference the field is uniform: r = t and φ = π.
- BiMFN. Motion features of both frames, warped by the previous level’s flows, with their 9x9 cost
volumes, the flows and the occlusion, form a motion feature that the embedded r and φ modulate; it predicts the residual of both flows at 1/4 size.
- CAUN. Context features warped by those flows predict convex-combination weights upsampling the flows
to 1/2 and full size.
- Synthesis. A U-Net over the warped frames, occlusion and warped context predicts a residual and the
occlusion logits that blend the two warped frames.
The levels run coarse-to-fine over an anti-aliased image pyramid of frames normalized by their joint mean and standard deviation and zero-padded on the right and bottom to multiples of 2^(levels + 1).
Training follows cfgs/bim_vfi.yaml with knowledge distillation for VFI-centric flow (KDVCF): a teacher that sees the ground-truth frame estimates the flows to it, whose ratio and angle feed the student’s BiMFN. The loss sums the student’s multi-level Charbonnier and ternary terms, the teacher’s photometric terms, the distillation of the teacher’s flow residuals, the zero-flow regularization of the teacher’s frame-side flows and the edge-aware smoothness of its flows. AdamW (weight decay 1e-4) follows a cosine one-cycle schedule peaking at 1.5e-4 after 1% of training, with gradients clipped to norm 1. The paper’s text instead reports a 1e-4 initial rate under cosine annealing at batch 32; the release’s configuration is the one that trained its checkpoints. The library’s one-cycle schedule warms up linearly where torch’s cosine-annealed one-cycle warms up along a cosine; the two meet at both ends of the 1% warm-up.
Constructors
| Constructor | Summary |
|---|---|
BiMVFI(NeuralNetworkArchitecture<T> architecture, BiMVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a BiM-VFI model in native training mode. |
BiMVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, BiMVFIOptions options) |
Creates a BiM-VFI model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |