FLAVR<T>

Models & Types · AiDotNet.Video.FrameInterpolation

FLAVR: flow-agnostic video representations for fast frame interpolation (Kalluri et al., WACV 2023).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → FLAVR<T>

For Beginners

FLAVR never estimates motion. It stacks four frames into a short clip, runs 3D (space and time) convolutions over it in a U-Net, and directly paints the in-between frame or frames.

How It Works

The network follows the release (tarun005/FLAVR, model/FLAVR_arch.py and model/resnet_3D.py): the context frames are stacked into a clip and its per-channel mean subtracted; an R3D-18 encoder with feature gating and no batch norm (a (3, 7, 7) stem at half size, then four stages, the middle two halving the size, time never strided) feeds a 3D U-Net decoder of gated convolutions and (3, 4, 4) transposed convolutions joined to the encoder by concatenation. The time steps are then stacked into channels, fused by a 1x1 convolution and mapped by a reflection-padded 7x7 convolution to every output frame, to which the mean is added back. Frames whose sides are not multiples of 8 are zero-padded on the right and bottom after the mean is subtracted.

The input is the window of NumInputFrames frames stacked along the channels, earliest first ([3·frames, H, W] or batched); the output is the NumOutputFrames evenly spaced frames in the middle gap, stacked the same way. A frame pair alone is accepted as the window with each frame repeated over its half, which is what Interpolate passes.

Training follows main.py: Adam (betas 0.9 and 0.99) at 2e-4 on the L1 loss over every output frame, the rate halved by ReduceLROnPlateau (patience 5) on the validation loss once per epoch.

Constructors

Constructor Summary
FLAVR(NeuralNetworkArchitecture<T> architecture, FLAVROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a FLAVR model in native training mode.
FLAVR(NeuralNetworkArchitecture<T> architecture, string modelPath, FLAVROptions options) Creates a FLAVR model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)
void EndEpoch(double validationLoss) Ends a training epoch with its validation loss, the release’s scheduler.step(test_loss): ReduceLROnPlateau multiplies the rate by PlateauFactor once the loss has not improved for PlateauPatience epochs.