VFIformerOptions

Options & Configuration · AiDotNet.Video.Options

Configuration options for VFIformer, video frame interpolation with a transformer (Lu et al., CVPR 2022).

Inheritance: ModelOptions → NeuralNetworkOptions → VFIformerOptions

For Beginners

VFIformer first estimates how pixels move between the two frames, warps both frames to the middle, and then uses a transformer, which compares each small window of the image with its surroundings at a coarser scale, to fix the places where the warping went wrong.

How It Works

The defaults are the release’s VFIformer (dvlab-research/VFIformer: models/archs/VFIformer_arch.py, transformer_layers.py, models/trainer.py): IFNet’s blocks of widths 240, 150 and 90, the width-24 flow refinement, and the transformer of width 160 with 8x8 windows, two heads, MLP ratio 2 and two RTFL groups of 3, 3, 3 and 1 TFLs at full, 1/2, 1/4 and 1/8 size, the second group at the first three sizes attending across scales. Training follows the release: AdamW at 1e-4 with a weight decay of 1e-4, a 2,000-step linear warmup and a cosine to zero over 300 epochs, on the L1 plus ternary loss, plus 0.01 times the end-point error of IFNet’s flows against precomputed flows when they are given.

Constructors

Constructor Summary
VFIformerOptions() Initializes a new instance with default values.
VFIformerOptions(VFIformerOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX runtime options.
bool[][] CrossScaleAttention { get; set; } Gets or sets, per scale (full size first) and per RTFL group, whether its TFLs also attend to the coarser windows: the second group at the first three scales.
double DropPathRate { get; set; } Gets or sets the stochastic depth rate, rising linearly over each scale’s TFLs (0.1).
double FlowWeight { get; set; } Gets or sets the flow end-point error’s weight when flows are given (lambda_flow, 0.01).
double L1Weight { get; set; } Gets or sets the L1 loss’s weight (lambda_l1, 1).
double LearningRate { get; set; } Gets or sets AdamW’s peak learning rate (1e-4).
double MlpRatio { get; set; } Gets or sets the MLP’s hidden width as a multiple of EmbedDim (2).
double TernaryWeight { get; set; } Gets or sets the ternary census loss’s weight (lambda_ter, 1).
double WeightDecay { get; set; } Gets or sets AdamW’s decoupled weight decay (1e-4).
int EmbedDim { get; set; } Gets or sets the transformer’s width (embed_dim, 160).
int NumHeads { get; set; } Gets or sets the attention heads (2), dividing EmbedDim.
int RefineChannels { get; set; } Gets or sets the flow refinement’s base width c (24); the context runs from c to 8c.
int TotalSteps { get; set; } Gets or sets the step at which the cosine reaches zero: 300 epochs of Vimeo-90K’s 51,312 triplets at the release’s batch of 24 over 8 GPUs, 2,138 steps per epoch (641,400).
int WarmupSteps { get; set; } Gets or sets the steps over which the rate rises linearly from zero (2,000).
int WindowSize { get; set; } Gets or sets the attention window side (8), even and a multiple of 4; frame sides are padded to 8·window.
int[] Depths { get; set; } Gets or sets the TFLs in each of a scale’s two RTFL groups, full size first (3, 3, 3, 1).
int[] FlowWidths { get; set; } Gets or sets IFNet’s three block widths, coarsest first (240, 150, 90); each is even.
string ModelPath { get; set; } Gets or sets the path to the ONNX model file.