BiMVFI<T>

Models & Types · AiDotNet.Video.FrameInterpolation

BiM-VFI: bidirectional motion field-guided frame interpolation for video with non-uniform motions (Seo, Oh and Kim, CVPR 2025).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → BiMVFI<T>

For Beginners

BiM-VFI describes how each pixel moves between the frames by a speed ratio and an angle, estimates the motion from the in-between moment back to each frame from coarse to fine, and blends the two warped frames into the in-between frame.

How It Works

The network follows the release (KAIST-VICLab/BiM-VFI, modules/components/bim_vfi):

  • Bidirectional motion field (BiM). The motion from the intermediate time to the two frames is described

by the ratio r of their magnitudes and the angle φ between them. At inference the field is uniform: r = t and φ = π.

  • BiMFN. Motion features of both frames, warped by the previous level’s flows, with their 9x9 cost

volumes, the flows and the occlusion, form a motion feature that the embedded r and φ modulate; it predicts the residual of both flows at 1/4 size.

  • CAUN. Context features warped by those flows predict convex-combination weights upsampling the flows

to 1/2 and full size.

  • Synthesis. A U-Net over the warped frames, occlusion and warped context predicts a residual and the

occlusion logits that blend the two warped frames.

The levels run coarse-to-fine over an anti-aliased image pyramid of frames normalized by their joint mean and standard deviation and zero-padded on the right and bottom to multiples of 2^(levels + 1).

Training follows cfgs/bim_vfi.yaml with knowledge distillation for VFI-centric flow (KDVCF): a teacher that sees the ground-truth frame estimates the flows to it, whose ratio and angle feed the student’s BiMFN. The loss sums the student’s multi-level Charbonnier and ternary terms, the teacher’s photometric terms, the distillation of the teacher’s flow residuals, the zero-flow regularization of the teacher’s frame-side flows and the edge-aware smoothness of its flows. AdamW (weight decay 1e-4) follows a cosine one-cycle schedule peaking at 1.5e-4 after 1% of training, with gradients clipped to norm 1. The paper’s text instead reports a 1e-4 initial rate under cosine annealing at batch 32; the release’s configuration is the one that trained its checkpoints. The library’s one-cycle schedule warms up linearly where torch’s cosine-annealed one-cycle warms up along a cosine; the two meet at both ends of the 1% warm-up.

Constructors

Constructor Summary
BiMVFI(NeuralNetworkArchitecture<T> architecture, BiMVFIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a BiM-VFI model in native training mode.
BiMVFI(NeuralNetworkArchitecture<T> architecture, string modelPath, BiMVFIOptions options) Creates a BiM-VFI model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)