FiganMultiScaleFlow<T>

Models & Types · AiDotNet.Video.FrameInterpolation

FIGAN’s multi-scale residual flow estimator and its occlusion-aware bidirectional synthesis.

For Beginners

To invent a frame between two others you first guess how things moved, starting from a blurry small version and sharpening the guess step by step. Then you drag both frames toward the middle and, pixel by pixel, decide which one to believe — because things hidden in one frame may be visible in the other.

How It Works

van Amersfoort et al., arXiv:1711.06045. The paper estimates flow coarse-to-fine over J = 3 scales, refining the upsampled coarser estimate at each finer level:

Gamma carries THREE channels, not two. Table 1 gives every flow module N_o = 3, and the synthesis needs a per-pixel occlusion weight W alongside the two displacement components — so Gamma = (dx, dy, W). The residual module’s N_i = 9 confirms it: the two warped RGB frames (3 + 3) plus the current 3-channel Gamma. Reading Gamma as pure 2-channel flow leaves the input widths unexplainable and W with nowhere to come from.

The tanh is on the SUM, not the residual.tanh(Gamma + Gamma_res) bounds the refined estimate; applying it to the residual alone would leave the accumulated flow unbounded across scales, which is what the bound exists to prevent.

Both directions are warped and blended by a LEARNED W. The mid-frame is not an average: I_0 is warped forward by -Delta, I_1 backward by +Delta, and W decides per pixel which to trust — that is how occlusions are handled. A fixed 0.5 blend is the naive baseline the paper improves on.

Constructors

Constructor Summary
FiganMultiScaleFlow(int scales) Creates the estimator.

Properties

Property Summary
int Scales { get; } Gets the number of scales J.

Methods

Method Summary
Tensor<T> Downsample(Tensor<T> frame, int level) Downsamples by successive halving, D^level.
Tensor<T> Refine(Tensor<T> upsampledFlow, Tensor<T> residual) Applies the paper’s refinement to an upsampled coarser estimate: tanh(Gamma + Gamma_res).
Tensor<T> Synthesise(Tensor<T> frame0, Tensor<T> frame1, Tensor<T> flow) The paper’s synthesis: W o I_0(-Delta) + (1 - W) o I_1(Delta).
Tensor<T> UpsampleFlow(Tensor<T> flow, int targetHeight, int targetWidth) Upsamples a flow field by 2x, U, scaling the displacement channels with it.
Tensor<T> Warp(Tensor<T> frame, Tensor<T> flow, int direction) Warps frame by direction times the flow, with bilinear sampling.

Fields

Field Summary
int FlowChannels Channels in Gamma: two displacement components plus the occlusion weight.