InterpAnyClearer<T>

Models & Types · AiDotNet.Video.FrameInterpolation

InterpAny-Clearer: distance indexing and iterative reference-based estimation for clearer anytime video frame interpolation (Zhong et al., ECCV 2024), on its release’s DR-RIFE.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → InterpAnyClearer<T>

For Beginners

Ordinary interpolation is told only “halfway in time”, which is ambiguous when objects speed up or slow down, so the result blurs. This model is instead told, for every pixel, how far along its path the object should be. It reaches that position in a few small steps, each starting from the frame the previous step produced, which keeps the result sharp.

How It Works

The network follows the release’s DR-RIFE (zzh-tech/InterpAny-Clearer, models/DI-RIFE: IFNet_sdi_recur.py, refine.py, RIFE_sdi_recur.py). Instead of a time, every pixel is given a distance index: how far along its path from frame 0 to frame 1 it should be. RIFE’s three IFBlocks estimate the flows to both frames and a blending mask coarse to fine, each reading the index map with the reference’s index and the reference frame; a Contextnet-U-Net adds a residual. At inference the target index is reached in Iterations steps from frame 0, each step’s frame the next step’s reference. Interpolate uses a uniform index (equal to the time), and InterpolateWithDistanceMap takes a per-pixel one. Sides are zero-padded on the right and bottom to multiples of 32, as the release pads.

Training follows the release (RIFE_sdi_recur.py, train_sdi_m_mask_recur.py): AdamW (weight decay 1e-3) with a 2,000-step linear warmup to 3e-4 and a cosine to 3e-6, on the Laplacian loss of the refined frame, plus that of the privileged teacher (which also reads the ground truth) and 0.002 times the distillation of the students’ flows toward the teacher’s where the teacher’s frame is better. A training input is the pair [6, H, W], optionally followed by the target’s distance index (one channel) or by that, the reference’s index and the reference frame (five channels). The release computes the indices from RAFT flows offline; without them the index is 0.5 and the reference is frame 0 with index 0.

Constructors

Constructor Summary
InterpAnyClearer(NeuralNetworkArchitecture<T> architecture, InterpAnyClearerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an InterpAny-Clearer model in native training mode.
InterpAnyClearer(NeuralNetworkArchitecture<T> architecture, string modelPath, InterpAnyClearerOptions options) Creates an InterpAny-Clearer model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)
Tensor<T> InterpolateWithDistanceMap(Tensor<T> frame0, Tensor<T> frame1, Tensor<T> distanceMap) The frame at a per-pixel distance index: for each pixel, the fraction of its path from frame0 to frame1 it should have travelled (0 at frame 0, 1 at frame 1).