RealViformer<T>

Models & Types · AiDotNet.Video.Enhancement

RealViformer: a recurrent transformer for real-world video super-resolution built on channel attention.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RealViformer<T>

For Beginners

Real videos are noisy and compressed, and attention that compares positions tends to copy those artifacts from frame to frame. RealViformer instead compares feature channels, which the authors found far less sensitive to artifacts, and learns to turn down channels that look unreliable.

How It Works

The network is the authors’ RealViformer for x4 real-world video super-resolution (Zhang and Yao, “RealViformer: Investigating Attention for Real-World Video Super-Resolution”, ECCV 2024; Yuehan717/RealViformer, archs/realviformer_arch.py and the configuration in inference_realviformer.py). A unidirectional recurrent loop warps the hidden state to each frame with SPyNet flow, merges it with the frame’s shallow features through channel cross- attention (the CAF), and reconstructs the result with a three-level Restormer encoder-decoder ([2, 3, 4] blocks with [48, 96, 192] channels and one refinement block) whose transposed attention is squeezed by 4 and re-weighted by a learned channel mask (the ICA). A 1x1 convolution makes the next hidden state, and x4 pixel-shuffle upsampling adds to the bilinearly upsampled frame.

Training follows Sec. 4.4: 15-frame clips, SPyNet frozen for the first 5,000 iterations, then 300,000 iterations of Charbonnier plus SSIM loss and another 130,000 iterations adding perceptual and GAN losses, weighted 1, 0.001, 1 and 0.005, with RealBasicVSR’s VGG19 perceptual loss, U-Net discriminator and vanilla GAN loss. The optimizer settings are in the unreleased supplementary material, so the rates are RealBasicVSR’s, whose losses the paper says it follows: Adam (0.9, 0.99) at 1e-4, then 5e-5 for the generator and 1e-4 for the discriminator. The degradations (blur, noise, JPEG and video compression) are data preparation and are not part of the model. The release’s SPyNet is loaded pretrained and starts from PyTorch’s default initialization here; its GELU is the erf form and the library’s the tanh approximation.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new RealViformer<float>(arch, new RealViformerOptions());
model.Train(degradedClip, cleanClip);        // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedClip);

Constructors

Constructor Summary
RealViformer(NeuralNetworkArchitecture<T> architecture, RealViformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a RealViformer model in native training mode.
RealViformer(NeuralNetworkArchitecture<T> architecture, string modelPath, RealViformerOptions options) Creates a RealViformer model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)