FiganMultiScaleFlow<T>
Models & Types · AiDotNet.Video.FrameInterpolation
FIGAN’s multi-scale residual flow estimator and its occlusion-aware bidirectional synthesis.
For Beginners
To invent a frame between two others you first guess how things moved, starting from a blurry small version and sharpening the guess step by step. Then you drag both frames toward the middle and, pixel by pixel, decide which one to believe — because things hidden in one frame may be visible in the other.
How It Works
van Amersfoort et al., arXiv:1711.06045. The paper estimates flow coarse-to-fine over
J = 3 scales, refining the upsampled coarser estimate at each finer level:
Gamma carries THREE channels, not two. Table 1 gives every flow module N_o = 3, and
the synthesis needs a per-pixel occlusion weight W alongside the two displacement
components — so Gamma = (dx, dy, W). The residual module’s N_i = 9 confirms it: the
two warped RGB frames (3 + 3) plus the current 3-channel Gamma. Reading Gamma as pure 2-channel
flow leaves the input widths unexplainable and W with nowhere to come from.
The tanh is on the SUM, not the residual.tanh(Gamma + Gamma_res) bounds the refined
estimate; applying it to the residual alone would leave the accumulated flow unbounded across
scales, which is what the bound exists to prevent.
Both directions are warped and blended by a LEARNED W. The mid-frame is not an average:
I_0 is warped forward by -Delta, I_1 backward by +Delta, and W decides
per pixel which to trust — that is how occlusions are handled. A fixed 0.5 blend is the naive
baseline the paper improves on.
Constructors
| Constructor | Summary |
|---|---|
FiganMultiScaleFlow(int scales) |
Creates the estimator. |
Properties
| Property | Summary |
|---|---|
int Scales { get; } |
Gets the number of scales J. |
Methods
| Method | Summary |
|---|---|
Tensor<T> Downsample(Tensor<T> frame, int level) |
Downsamples by successive halving, D^level. |
Tensor<T> Refine(Tensor<T> upsampledFlow, Tensor<T> residual) |
Applies the paper’s refinement to an upsampled coarser estimate: tanh(Gamma + Gamma_res). |
Tensor<T> Synthesise(Tensor<T> frame0, Tensor<T> frame1, Tensor<T> flow) |
The paper’s synthesis: W o I_0(-Delta) + (1 - W) o I_1(Delta). |
Tensor<T> UpsampleFlow(Tensor<T> flow, int targetHeight, int targetWidth) |
Upsamples a flow field by 2x, U, scaling the displacement channels with it. |
Tensor<T> Warp(Tensor<T> frame, Tensor<T> flow, int direction) |
Warps frame by direction times the flow, with bilinear sampling. |
Fields
| Field | Summary |
|---|---|
int FlowChannels |
Channels in Gamma: two displacement components plus the occlusion weight. |