FLAVR<T>
Models & Types · AiDotNet.Video.FrameInterpolation
FLAVR: flow-agnostic video representations for fast frame interpolation (Kalluri et al., WACV 2023).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → FLAVR<T>
For Beginners
FLAVR never estimates motion. It stacks four frames into a short clip, runs 3D (space and time) convolutions over it in a U-Net, and directly paints the in-between frame or frames.
How It Works
The network follows the release (tarun005/FLAVR, model/FLAVR_arch.py and model/resnet_3D.py): the context frames are stacked into a clip and its per-channel mean subtracted; an R3D-18 encoder with feature gating and no batch norm (a (3, 7, 7) stem at half size, then four stages, the middle two halving the size, time never strided) feeds a 3D U-Net decoder of gated convolutions and (3, 4, 4) transposed convolutions joined to the encoder by concatenation. The time steps are then stacked into channels, fused by a 1x1 convolution and mapped by a reflection-padded 7x7 convolution to every output frame, to which the mean is added back. Frames whose sides are not multiples of 8 are zero-padded on the right and bottom after the mean is subtracted.
The input is the window of NumInputFrames frames stacked along the channels, earliest
first ([3·frames, H, W] or batched); the output is the NumOutputFrames evenly spaced
frames in the middle gap, stacked the same way. A frame pair alone is accepted as the window with each frame
repeated over its half, which is what Interpolate passes.
Training follows main.py: Adam (betas 0.9 and 0.99) at 2e-4 on the L1 loss over every output frame, the rate halved by ReduceLROnPlateau (patience 5) on the validation loss once per epoch.
Constructors
| Constructor | Summary |
|---|---|
FLAVR(NeuralNetworkArchitecture<T> architecture, FLAVROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a FLAVR model in native training mode. |
FLAVR(NeuralNetworkArchitecture<T> architecture, string modelPath, FLAVROptions options) |
Creates a FLAVR model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |
|
void EndEpoch(double validationLoss) |
Ends a training epoch with its validation loss, the release’s scheduler.step(test_loss): ReduceLROnPlateau multiplies the rate by PlateauFactor once the loss has not improved for PlateauPatience epochs. |