DualXVSR<T>

Models & Types · AiDotNet.Video.Enhancement

DualX-VSR: a dual axial spatial x temporal transformer for real-world video super-resolution without motion compensation.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → DualXVSR<T>

For Beginners

Most video upscalers first work out how things moved between frames. DualX-VSR skips that: its attention looks along a column across all frames, then along a row across all frames, so every position can draw on matching content anywhere in the clip, like solving a crossword by reading down and across.

How It Works

The network is Cao et al.’s DualX-VSR (arXiv 2506.04830, 2025), a ViViT-based model with no optical flow, alignment or recurrence. Each frame passes a 3x3 convolution to 64 channels, a two-block temporal transformer mixes every pixel’s frames, and 2x2 patches become 1,280-wide tokens. Six vertical-temporal blocks attend over all height x time positions of each token column, then six horizontal-temporal blocks over all width x time positions of each row, with rotary positions (RoPE, base 10,000). The tokens are decoded back to pixels, a second two-block temporal transformer runs, and per-frame convolution and pixel shuffle upsample by four.

The paper releases no code, so what it leaves open follows the ViViT implementation it builds on (vit-pytorch): pre-norm residual blocks with LayerNorm, bias-free q/k/v projections to 12 heads of 64 channels, GELU MLPs, a final norm per transformer and the LayerNorm-Linear-LayerNorm patch embedding. With these choices the default network has 127.8M parameters against the paper’s 127.95M. RoPE over a height (or width) x time sequence gives each axis half of a head’s channels; the upsampler is BasicVSR’s (pixel shuffle x2 stages with LeakyReLU, conv_hr, conv_last).

Training follows Sec. 5.1 and App. C.1: AdamW at 5e-5 on Charbonnier loss for 500,000 pretraining and 150,000 fine-tuning iterations (bicubic, then RealBasicVSR’s first-order degradations: data preparation, not the model), then 25,000 iterations with Charbonnier, perceptual and GAN losses weighted 0.01, 1 and 0.005 following BasicVSR, at 1e-5 for the generator and RealBasicVSR’s spectral-norm U-Net discriminator, with its VGG19 perceptual loss and vanilla GAN loss. AdamW’s betas and decay are unstated, so they are PyTorch’s defaults.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 16, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new DualXVSR<float>(arch, new DualXVSROptions());
model.Train(degradedClip, cleanClip);        // [16, 3, 64, 64] -> [16, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedClip);

Constructors

Constructor Summary
DualXVSR(NeuralNetworkArchitecture<T> architecture, DualXVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a DualX-VSR model in native training mode.
DualXVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, DualXVSROptions options) Creates a DualX-VSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)