VFIformerOptions
Options & Configuration · AiDotNet.Video.Options
Configuration options for VFIformer, video frame interpolation with a
transformer (Lu et al., CVPR 2022).
Inheritance: ModelOptions → NeuralNetworkOptions → VFIformerOptions
For Beginners
VFIformer first estimates how pixels move between the two frames, warps both frames to the middle, and then uses a transformer, which compares each small window of the image with its surroundings at a coarser scale, to fix the places where the warping went wrong.
How It Works
The defaults are the release’s VFIformer (dvlab-research/VFIformer: models/archs/VFIformer_arch.py, transformer_layers.py, models/trainer.py): IFNet’s blocks of widths 240, 150 and 90, the width-24 flow refinement, and the transformer of width 160 with 8x8 windows, two heads, MLP ratio 2 and two RTFL groups of 3, 3, 3 and 1 TFLs at full, 1/2, 1/4 and 1/8 size, the second group at the first three sizes attending across scales. Training follows the release: AdamW at 1e-4 with a weight decay of 1e-4, a 2,000-step linear warmup and a cosine to zero over 300 epochs, on the L1 plus ternary loss, plus 0.01 times the end-point error of IFNet’s flows against precomputed flows when they are given.
Constructors
| Constructor | Summary |
|---|---|
VFIformerOptions() |
Initializes a new instance with default values. |
VFIformerOptions(VFIformerOptions other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
Gets or sets the ONNX runtime options. |
bool[][] CrossScaleAttention { get; set; } |
Gets or sets, per scale (full size first) and per RTFL group, whether its TFLs also attend to the coarser windows: the second group at the first three scales. |
double DropPathRate { get; set; } |
Gets or sets the stochastic depth rate, rising linearly over each scale’s TFLs (0.1). |
double FlowWeight { get; set; } |
Gets or sets the flow end-point error’s weight when flows are given (lambda_flow, 0.01). |
double L1Weight { get; set; } |
Gets or sets the L1 loss’s weight (lambda_l1, 1). |
double LearningRate { get; set; } |
Gets or sets AdamW’s peak learning rate (1e-4). |
double MlpRatio { get; set; } |
Gets or sets the MLP’s hidden width as a multiple of EmbedDim (2). |
double TernaryWeight { get; set; } |
Gets or sets the ternary census loss’s weight (lambda_ter, 1). |
double WeightDecay { get; set; } |
Gets or sets AdamW’s decoupled weight decay (1e-4). |
int EmbedDim { get; set; } |
Gets or sets the transformer’s width (embed_dim, 160). |
int NumHeads { get; set; } |
Gets or sets the attention heads (2), dividing EmbedDim. |
int RefineChannels { get; set; } |
Gets or sets the flow refinement’s base width c (24); the context runs from c to 8c. |
int TotalSteps { get; set; } |
Gets or sets the step at which the cosine reaches zero: 300 epochs of Vimeo-90K’s 51,312 triplets at the release’s batch of 24 over 8 GPUs, 2,138 steps per epoch (641,400). |
int WarmupSteps { get; set; } |
Gets or sets the steps over which the rate rises linearly from zero (2,000). |
int WindowSize { get; set; } |
Gets or sets the attention window side (8), even and a multiple of 4; frame sides are padded to 8·window. |
int[] Depths { get; set; } |
Gets or sets the TFLs in each of a scale’s two RTFL groups, full size first (3, 3, 3, 1). |
int[] FlowWidths { get; set; } |
Gets or sets IFNet’s three block widths, coarsest first (240, 150, 90); each is even. |
string ModelPath { get; set; } |
Gets or sets the path to the ONNX model file. |