RealViformerOptions

Options & Configuration · AiDotNet.Video.Options

Configuration options for RealViformer, the channel-attention recurrent transformer for real-world video super-resolution.

Inheritance: ModelOptions → NeuralNetworkOptions → RealViformerOptions

For Beginners

RealViformer upscales real, messy video four times. These settings control how wide and deep the network is and how it is trained: first to match the clean frames, then to also look realistic.

How It Works

Defaults follow Zhang and Yao, ECCV 2024, Sec. 4.4, and the authors’ release (Yuehan717/RealViformer, inference_realviformer.py): 48 channels, [2, 3, 4] transformer blocks over three levels plus one refinement block, [1, 2, 4] heads, GDFN expansion 2.66, a two-head CAF, attention squeezed by 4 with the channel mask, 15-frame training clips, SPyNet frozen for 5,000 iterations, 300,000 iterations of Charbonnier + SSIM and then perceptual and GAN losses (weights 1, 0.001, 1, 0.005) with RealBasicVSR’s discriminator. The optimizer is not in the public paper, so the rates are RealBasicVSR’s: Adam (0.9, 0.99) at 1e-4, then 5e-5 for the generator and 1e-4 for the discriminator.

Constructors

Constructor Summary
RealViformerOptions() Initializes a new instance with default values.
RealViformerOptions(RealViformerOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX runtime options.
bool ChannelMask { get; set; } Gets or sets whether the attention re-weights its channels with the learned mask (true).
double AdversarialLearningRate { get; set; } Gets or sets the second stage’s generator Adam rate (5e-5).
double CharbonnierEpsilon { get; set; } Gets or sets the Charbonnier ε (1e-6, i.e.
double CharbonnierWeight { get; set; } Gets or sets the Charbonnier loss weight (1).
double DiscriminatorLearningRate { get; set; } Gets or sets the discriminator’s Adam rate (1e-4).
double FfnExpansionFactor { get; set; } Gets or sets the GDFN’s hidden width as a multiple of the channels (2.66).
double GANWeight { get; set; } Gets or sets the GAN loss weight in the second stage (0.005).
double LearningRate { get; set; } Gets or sets the first stage’s Adam rate (1e-4).
double PerceptualWeight { get; set; } Gets or sets the perceptual loss weight in the second stage (1; 0 leaves the VGG19 out).
double SsimWeight { get; set; } Gets or sets the SSIM loss weight (0.001).
double? EmaMomentum { get; set; } Gets or sets the momentum of a moving average of the weights used for inference; null (the paper reports none) uses the trained weights.
int DiscriminatorChannels { get; set; } Gets or sets the discriminator’s base width (64, RealBasicVSR’s U-Net discriminator).
int FlowFreezeIterations { get; set; } Gets or sets the iterations SPyNet stays fixed at the start of training (5,000).
int MergeHeads { get; set; } Gets or sets the heads of the CAF that merges the hidden state (2).
int NumFeatures { get; set; } Gets or sets the first level’s channels (48; the levels below have 96 and 192).
int NumFrames { get; set; } Gets or sets the frames of a training clip (15).
int ReconstructionIterations { get; set; } Gets or sets the iterations of the Charbonnier + SSIM stage before perceptual and GAN losses join (300,000).
int SqueezeFactor { get; set; } Gets or sets the factor the attention squeezes channels by (4; 1 disables the squeeze).
int[] Heads { get; set; } Gets or sets the attention heads of levels 1-3 ([1, 2, 4]).
int[] NumBlocks { get; set; } Gets or sets the transformer blocks of levels 1-3 and the refinement ([2, 3, 4, 1]).
string ModelPath { get; set; } Gets or sets the path to the ONNX model file.