IFRNet<T>
Models & Types · AiDotNet.Video.FrameInterpolation
IFRNet: intermediate feature refine network for efficient frame interpolation (Kong et al., CVPR 2022).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → IFRNet<T>
For Beginners
IFRNet encodes both frames into a four-level pyramid, then decodes from the coarsest level up, each decoder refining the two motions from the in-between frame and a feature of the in-between frame itself; the last one also predicts how to blend the two warped frames and a correction.
How It Works
The network follows the release (ltkong218/IFRNet, models/IFRNet.py, IFRNet_L.py and IFRNet_S.py): a shared four-level encoder (1/2 to 1/16 size) and four decoders from 1/16 size. The first decoder reads both frames’ coarsest features and the time t; each later one reads the previous intermediate feature, both frames’ features warped by the current flows, and the flows, and refines the flows (added to the 2x-upsampled ones) and the intermediate feature. The last also predicts a blending mask and a residual. The mean of both frames is subtracted and restored, and frames are zero-padded on the right and bottom to multiples of 16, as the release’s benchmarks pad. Any time in [0, 1] can be synthesized.
Training follows train_vimeo90k.py at the midpoint: AdamW (no weight decay) with cosine decay from 1e-4 to 1e-5 per iteration, on the Charbonnier and ternary reconstruction loss plus 0.01 times the geometry consistency between each decoder’s intermediate feature and the encoder’s features of the ground-truth frame. The release’s task-oriented flow distillation needs pseudo ground-truth flows from a pretrained LiteFlowNet that a frame-pair to frame sample does not carry, so it is not applied.
Constructors
| Constructor | Summary |
|---|---|
IFRNet(NeuralNetworkArchitecture<T> architecture, IFRNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an IFRNet model in native training mode. |
IFRNet(NeuralNetworkArchitecture<T> architecture, string modelPath, IFRNetOptions options) |
Creates an IFRNet model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |