M2M<T>

Models & Types · AiDotNet.Video.FrameInterpolation

M2M: many-to-many splatting for efficient video frame interpolation (Hu, Niklaus, Sclaroff and Saenko, CVPR 2022).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → M2M<T>

For Beginners

M2M estimates rough motion once, then refines it into several motion guesses for every pixel. Each pixel is pushed forward along all of its guesses to the in-between time, and where several pixels land on the same spot they are blended by how reliable each one is. There is no heavy synthesis network, so it is fast.

How It Works

The network follows the release’s M2M-PWC (feinanshan/M2M_VFI, m2m.py). Both frames are standardized by their joint mean and standard deviation. PWC-Net estimates the bidirectional flows at 1/ratio size; the Motion Refinement Network upsamples them and, from both frames’ feature pyramids warped toward each other and a rank-16 attention cube at 1/16 size, adds four flow residuals per pixel each way and predicts a photometric weight. Every pixel of each frame is splatted along all four of its flows to time t, weighted by its distance in time and exp(α·(1 − weight·photometric error)²), and the sum is divided by the summed weights; pixels nothing reaches take the time-weighted blend. Sides are replicate-padded on the right and bottom to multiples of 16·ratio, as the release pads. Any time in [0, 1] can be synthesized from one motion estimate.

Training follows the release (train.py, optimizer.py, configs/vimeo90k.yml): AdamW at 1e-4 with a weight decay of 1e-4, a 2,000-step linear warmup from zero and cosine decay to zero at 900,000 steps, gradients clipped to norm 1, on the Charbonnier plus ternary census loss at t = 0.5. The release loads a pretrained PWC-Net and trains it at a learning rate of zero; FlowLearningRateMultiplier sets that multiple.

Constructors

Constructor Summary
M2M(NeuralNetworkArchitecture<T> architecture, M2MOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an M2M model in native training mode.
M2M(NeuralNetworkArchitecture<T> architecture, string modelPath, M2MOptions options) Creates an M2M model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)